Catalog Factory

iceberg_ops.core.catalog

Catalog factory for PyIceberg.

CatalogFactory.build(kind, ...) is the single entry point for obtaining a catalog instance regardless of the backend. SQLite and in-memory backends are implemented natively; the remote backends (Glue, REST, Hive, S3 Tables) go through pyiceberg.catalog.load_catalog(), so they also pick up ~/.pyiceberg.yaml and PYICEBERG_CATALOG__<NAME>__* environment variables.

Bring Your Own Catalog (BYOC)

Every iceberg-ops API that takes a catalog accepts any PyIceberg Catalog subclass. CatalogFactory is a convenience helper — you are not required to use it:

from pyiceberg.catalog.glue import GlueCatalog
my_catalog = GlueCatalog("prod", region_name="us-east-1")

from iceberg_ops.runner import MigrationRunner
runner = MigrationRunner(my_catalog, namespace="sales")

Lazy imports

Backend-specific modules are imported through the _lazy_import helper inside _build_* methods rather than at module load time, so iceberg_ops.core.catalog always imports cleanly even when only a subset of the optional extras is installed.

Default local paths

When build("sqlite") is called without db_path / warehouse_dir, the catalog lives under ./local_data/ in the current working directory at call time (see default_local_paths()). LOCAL_DB_PATH and LOCAL_WH_DIR are those defaults as relative paths. Add local_data/ to .gitignore so generated files are not committed. The "memory" backend never uses ./local_data/: without warehouse_dir its data files go to a fresh temporary directory.

iceberg_ops.core.catalog.default_local_paths(base_dir=None)[source]

Return the default SQLite catalog paths under base_dir.

Parameters:

base_dir (str | PathLike[str] | None, default: None) – Directory that holds local_data/. Defaults to the current working directory, read when this function is called.

Returns:

Absolute (db_path, warehouse_dir): <base_dir>/local_data/iceberg_catalog.db and <base_dir>/local_data/iceberg_warehouse.

Return type:

tuple[str, str]

class iceberg_ops.core.catalog.CatalogFactory[source]

Bases: object

Factory that creates PyIceberg Catalog instances.

Supported backends

"sqlite"

Local file-backed SQLite SqlCatalog. Requires the sqlite extra (pyiceberg[sql-sqlite], which brings in SQLAlchemy).

"memory"

In-memory SQLite SqlCatalog (uri="sqlite://", no .db file). Catalog state is lost when the process exits; Parquet data files land on warehouse_dir, or in a fresh temporary directory when it is omitted. Pair with pytest’s tmp_path for fully isolated tests.

"glue"

AWS Glue Iceberg catalog. Requires the glue extra.

"rest"

REST-protocol Iceberg catalogs (Apache Polaris, Nessie, Lakekeeper, the AWS Lakehouse REST endpoint, …). PyIceberg’s REST client is built in, so no extra is needed.

"hive"

Hive Metastore Iceberg catalog. Requires the hive extra.

"s3tables"

AWS S3 Tables (REST + SigV4 under the hood). Requires the s3tables extra (which pulls in pyiceberg[rest-sigv4]). Pre-sets the SigV4 REST properties so callers only have to provide uri (REST endpoint), warehouse (bucket ARN) and region. S3 Tables enforces single-level namespaces — namespaces with dots are rejected by the server.

The remote backends are loaded with pyiceberg.catalog.load_catalog(), so the catalog’s entry in .pyiceberg.yaml and its PYICEBERG_CATALOG__<NAME>__* environment variables are merged in; explicit keyword arguments win.

Notes

To add a backend, implement a _build_<kind> static method and register it in the builders dict inside build().

static build(kind='sqlite', *, reset=False, db_path=None, warehouse_dir=None, **kwargs)[source]

Build and return a ready-to-use PyIceberg catalog.

Parameters:

  • kind (str, default: 'sqlite') – Backend identifier. Case-insensitive. Default "sqlite".

  • reset (bool, default: False) – Delete the existing catalog DB and warehouse directory before creating a fresh one. Only applies to the "sqlite" backend (file-backed); ignored for "memory" and the remote backends. Refuses to delete a filesystem root, the home directory, the working directory or one of its parents, and a db_path that is not an SQLite file. Destructive — use only in dev / test.

  • db_path (str | PathLike[str] | None, default: None) – SQLite catalog database path. Only used by "sqlite", where it defaults to ./local_data/iceberg_catalog.db under the current working directory at call time. Ignored for "memory" (no file is created). Must not contain ?.

  • warehouse_dir (str | PathLike[str] | None, default: None) – Root for Iceberg data files (Parquet / Avro). For "sqlite" it defaults to ./local_data/iceberg_warehouse under the current working directory; for "memory" to a new temporary directory. For the local backends it must be a local directory, not a URI, and its absolute path must not contain # or ? (the backend passes it to PyIceberg as an unescaped file:// URI). For the remote backends it sets the warehouse catalog property (a location or URI) unless warehouse is passed explicitly.

  • **kwargs (Any) – catalog_name names the catalog (for the remote backends this is the name looked up in .pyiceberg.yaml and the environment; it defaults to kind). Every other keyword is a catalog property passed to the backend (e.g. uri, region, glue.region). The local backends reject uri and warehouse; use db_path / warehouse_dir.

Returns:

A live PyIceberg catalog instance.

Return type:

Catalog

Raises:

  • NotImplementedError – If kind is not one of the supported backends.

  • ImportError – If the optional extra for kind is not installed.

  • ValueError – If db_path is given for a remote backend, if a local backend gets uri / warehouse, a URI as warehouse_dir, or a warehouse_dir / db_path holding a character its URI form cannot carry (# or ?), if reset targets an unsafe path, or if PyIceberg rejects the catalog properties (for example a missing uri).