gluecat
AWS Glue Data Catalog client for Apache Spark.
gluecat is a Spark catalog plugin, io.github.sdaberdaku.gluecat.GlueCatalog. Each Spark catalog configured with it
talks to one Glue Data Catalog, with its own catalog ID and its own credentials or IAM role. It loads, creates and
alters Delta tables there, reads other data source tables, and manages Glue databases. It works as a named catalog
and as spark_catalog. The supported API is the class io.github.sdaberdaku.gluecat.GlueCatalog, its configuration keys, the Glue entry format and the exception classes; other public classes are internals that may change between minor versions. It does not use the Hive metastore client, needs no patched Spark or Hive jars, and uses the
AWS SDK for Java v2.
API documentation: Scaladoc.
Why
Spark and Delta Lake have no Glue support of their own. Spark's built-in session catalog, spark_catalog, talks to
a Hive metastore, and Delta Lake works only through that catalog (DeltaCatalog wraps it). Reaching Glue at all means
installing AWS's Glue Data Catalog client for Hive, patched Hive jars and the AWS SDK v1 included, and plugging it into
that one session catalog. The result is one Glue catalog ID and one set of credentials for the whole Spark application.
When each customer, team or environment has its own AWS account, the S3 data can already live in each account, but every Delta table still has to be registered in one shared Glue catalog. A single Spark application, for example a Spark Connect server that runs jobs for several accounts, cannot register account A's tables in account A's Glue and account B's tables in account B's Glue.
Iceberg's GlueCatalog already gives each named catalog its own Glue catalog ID and IAM role. gluecat brings that
model to Glue tables in general and to Delta tables in particular. Its option names follow Iceberg's AWS properties.
Artifacts
One artifact per Spark minor version and Scala version, group io.github.sdaberdaku:
| Artifact | Spark | Scala | Java bytecode | JDKs tested | Delta tested |
|---|---|---|---|---|---|
gluecat-spark-3.5_2.12 |
3.5.3 and later 3.5 | 2.12 | 8 | 8, 11, 17 | 3.3.3 |
gluecat-spark-3.5_2.13 |
3.5.3 and later 3.5 | 2.13 | 8 | 8, 11, 17 | 3.3.3 |
gluecat-spark-4.0_2.13 |
4.0.x | 2.13 | 17 | 17, 21 | 4.0.1, 4.1.0, 4.2.0, 4.3.1, 4.4.1 |
gluecat-spark-4.1_2.13 |
4.1.x | 2.13 | 17 | 17, 21 | 4.1.0 and 4.2.0 (on Spark 4.1.1), 4.3.1, 4.4.1 |
gluecat-spark-4.2_2.13 |
4.2.x | 2.13 | 17 | 17, 21, 25 | 4.4.1 |
Spark 3.5.0 to 3.5.2 are not supported: they reject the tables this catalog returns for non-Delta data, and
initialize fails on them. matrix.json lists every Spark, Delta and JDK combination CI runs.
Delta is optional: without Delta on the classpath, the catalog serves non-Delta tables only.
Where to go next
- Quick start: dependencies and configuration.
- Configuration: every option, credentials, AWS SDK requirements.
- Operations: what works, for Delta and for other tables.
- Glue entry format: what the catalog writes to Glue and which readers it works with.
- How it works: why Delta creates run by path.
- Limitations.