Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Added

  • Project logo. The docs show it in the sidebar, in a light and a dark version, and use it as a favicon that follows the browser’s light or dark theme. Links to the docs get a preview image (Open Graph), and the README shows the logo next to its introduction.

0.1.0 - 2026-11-08

First public release of iceberg-ops: Alembic-style schema migrations for Apache Iceberg, plus a cross-table data mover, basic maintenance helpers and an experimental Apache Polaris policy executor.

Development builds from May 2026 were never published. The Changed and Fixed lists below describe how this release differs from them, for anyone who ran those builds from source.

Added

Schema migrations:

  • BaseMigration and MigrationContext for migrations/NNNN_*.py files with upgrade() / downgrade() and the pre_upgrade, post_upgrade, pre_downgrade and post_downgrade hooks. Context helpers: create_table_if_not_exists (with partition_spec, sort_order, properties and location, applied on create), drop_table_if_exists, load_table, add_column, enforce_column (with opt-in legacy_archival), enforce_rename, delete_column, update_partition_spec, replace_sort_order, set_clustering_keys, set_table_properties, unset_table_properties, set_snapshot_retention and set_compaction_target_size. Helpers take bare table names; the runner adds the namespace.

  • MigrationRunner with migrate, rollback, stamp, plan, status and history for one namespace at a time. Migration history is an Iceberg table, _iceberg_ops.migration_history, in the managed catalog; it keeps the latest status of each version per namespace.

  • iceberg_ops.runner.normalize_version: "2", "0002" and 2 name the same version.

  • The iceberg-ops CLI: init, revision, migrate, rollback, stamp, plan, status, current, history, migrate-data and run-policies, plus --version. Settings come from flags or from iceberg-ops.yaml (keys catalog, catalog_name, namespace, db_path, warehouse, migrations_dir).

  • --catalog-name NAME and the config key catalog_name open any catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* environment variables. migrate-data has --src-catalog-name / --dst-catalog-name and run-policies has --catalog-name.

  • Autogenerate (iceberg_ops.autogen): schema_diff_to_ops, spec_diff_to_ops, sort_order_diff_to_ops, properties_diff_to_ops, render_migration_body, render_migration_file and write_autogenerated_revision, plus iceberg-ops revision --autogenerate --target-table T with --target-schema-yaml FILE or --target-schema-arrow module:attr and repeatable --rename-hint old:new. The CLI diffs columns; the Python API also diffs partition spec, sort order and table properties. Rename hints are validated, and new columns keep their target doc.

  • YAML schema contracts (SchemaContract.from_yaml) accept the Iceberg primitive types, including decimal(P,S), fixed[N] and timestamp_ns, and are validated with clear errors.

  • Core helpers (iceberg_ops.core): CatalogFactory.build for the sqlite, memory, glue, rest, hive and s3tables kinds, default_local_paths(), Arrow-to-Iceberg type translation with is_allowed_promotion(old, new) (Arrow time, uuid, fixed-size binary, string_view, binary_view and dictionary types are mapped), schema diff, SchemaContract, SchemaEvolutionHelper, and snapshot, upsert, time-travel and field-history utilities.

Companion modules:

  • Cross-table data migration, experimental (iceberg_ops.data_migration, iceberg-ops migrate-data): SchemaReconciler, CheckpointStore and DataMigrator with append, upsert and overwrite_partition write modes, checkpoint resume, dry runs and row_count / partition_hash / sample verification. MigrationResult reports dest_snapshot_before (a rollback target), rows_inserted and rows_updated; CheckpointStore.job exposes the recorded job settings. Every snapshot the migrator writes carries iceberg-ops.migration-job and iceberg-ops.migration-batch summary properties.

  • Basic maintenance helpers for self-managed tables (iceberg_ops.maintenance): compact_small_files_streaming, expire_snapshots, remove_orphan_files and table_health_report. SnapshotExpiryResult.protected_ids lists snapshots kept because a branch or tag points at them; OrphanCleanupResult.failed_paths lists orphans whose deletion failed.

  • Apache Polaris policy executor, experimental (iceberg_ops.polaris, iceberg-ops run-policies): PolarisClient, Policy, parse_policy, fetch_table_policies, dispatch_policy, dispatch_policies and a handlers={...} hook for custom policy types. Only system.snapshot-expiry is executed.

  • iceberg_ops.testing: FakeResponse and FakeSession test doubles for driving the Polaris client without HTTP. They may change before 1.0.

Packaging and project:

  • Extras: sqlite, glue, hive, s3tables, transforms (needed to write to tables partitioned by bucket, truncate or time transforms) and polaris. rest and contracts are accepted but add nothing to the core install.

  • A py.typed marker, so type checkers use the package’s annotations.

  • Documentation at https://namitkewat.github.io/iceberg-ops/, including a compatibility page and plain-text llms.txt / llms-full.txt exports of the docs.

  • CONTRIBUTING.md, CODE_OF_CONDUCT.md (Contributor Covenant 3.0), SECURITY.md (GitHub private vulnerability reporting), ROADMAP.md, issue forms and a pull request template.

  • AGENTS.md, a guide for coding agents working on the repository, and agent-skills/iceberg-ops-migrations/SKILL.md, a skill that teaches a coding agent to write, review and run iceberg-ops migrations.

Changed

Packaging and CLI:

  • The command-line tool is iceberg-ops only, and its config file is iceberg-ops.yaml. The earlier short script name and its config file name were dropped: that name belongs to an unrelated project.

  • PyIceberg is pinned to >=0.12.0,<0.13 (was >=0.11.0,<0.12) and PyArrow to >=18. Python 3.14 is tested in CI. PyYAML is a core dependency, and tzdata is installed on Windows.

  • The [maintenance] extra was removed: it only installed boto3, which no module imported. [s3tables] installs pyiceberg[rest-sigv4]; [rest] and [polaris] no longer request a pyiceberg[rest] extra, which PyIceberg does not define.

  • Development status is Alpha.

  • Default local state: without db_path / warehouse_dir, CatalogFactory.build("sqlite") uses ./local_data/ in the current working directory at call time, and the CLI’s sqlite catalog uses local_data/ next to the config file (or in the current directory when there is none). CatalogFactory.build("memory") without warehouse_dir writes data files to a fresh temporary directory.

  • The glue, rest, hive and s3tables kinds are loaded through pyiceberg.catalog.load_catalog, so .pyiceberg.yaml and PYICEBERG_CATALOG__<KIND>__* variables apply; explicit arguments win. db_path with a remote kind raises ValueError, and the local kinds reject uri / warehouse keyword arguments.

  • The CLI no longer accepts the memory catalog kind, which forgot applied migrations after each command. Catalog-kind options are case-insensitive, and a typo is a usage error.

  • Relative paths in the config file (db_path, the sqlite warehouse, migrations_dir) resolve against the config file’s directory, and migrations/ defaults to sitting next to it. Paths given as flags stay relative to the current directory.

  • CLI exit codes: 2 for usage errors (bad flags or config, missing catalog, namespace or migrations directory, invalid --target / --steps); 1 for runtime failures, with a single Error: line. Set ICEBERG_OPS_TRACEBACK=1 to see the traceback.

  • A missing migrations directory is an error (FileNotFoundError from the runner, a usage error in the CLI) instead of reporting that all migrations are up to date. An explicit --config path that does not exist is a usage error, except for init.

Migrations:

  • A migration’s version must equal its file’s NNNN prefix, and two files with the same version are rejected. revision numbers new files by the same NNNN_<description>.py rule the runner uses.

  • plan, status and history no longer create the _iceberg_ops namespace or the history table; the first write creates them.

  • --target / target= is compared as a version number: 2 means 0002, and an unknown or malformed target raises ValueError (CLI: exit 2) before anything is applied. rollback(target="0000") rolls back every migration. rollback --steps must be 1 or more, defaults to 1 and cannot be combined with --target.

  • enforce_column adds a missing column, does nothing for an equivalent type (only a changed doc is written), updates a promotable primitive type in place, and otherwise raises pyiceberg.exceptions.ValidationError unless legacy_archival=True. Promotions follow Iceberg’s rules, so string-to-binary and decimal scale changes are rejected.

  • set_clustering_keys sets only the Iceberg sort order, in one commit. It no longer writes snowflake.cluster-by or delta.clustering table properties.

  • set_snapshot_retention logs a warning when the catalog is AWS S3 Tables, which manages snapshot retention itself.

  • ConcurrentMigrationError is defined in iceberg_ops.history and is still importable from iceberg_ops.runner.

  • revision never overwrites an existing migration file (exit 1). write_autogenerated_revision refuses to overwrite one unless overwrite=True, and always writes LF line endings.

Autogenerate:

  • A bare --target-table stays bare in the generated file, so the migration applies to whichever --namespace it runs against. A dotted name pins the file to that table and prints a warning.

  • A generated upgrade() or downgrade() raises NotImplementedError before any step runs when one of its steps cannot run or cannot be undone (a column drop, a type widening, a partition-field removal, a sort-order replacement, a property removal with an unknown old value). The generated calls are kept as comments.

  • Type-change downgrades are no longer generated as narrowing calls, since Iceberg only widens types; they raise NotImplementedError.

  • A non-strict SchemaContract (the default) keeps the table columns it does not list, and the CLI reports them. Set strict: true to drop them. Nullability and doc differences on existing columns are reported, not emitted.

Data migration and maintenance:

  • write_mode="overwrite_partition" needs a destination partitioned by identity transforms only, and the migration must fit in one commit (at most batch_rows source rows after the row filter). Anything else is rejected before any write.

  • Checkpoints record the job’s settings (source and destination table UUIDs, row filter, batch_rows, write mode, upsert keys and reconciler). Resuming with different settings, a recreated destination or a checkpoint without these settings raises CheckpointError, which is now a RuntimeError. A completed checkpoint is terminal: call CheckpointStore.reset() to run again.

  • MigrationResult.new_snapshots lists every destination snapshot the run created (up to three per upsert batch, or per 1,000-key sub-batch for composite keys), and CheckpointStore.mark_committed accepts a list of snapshot IDs.

  • DataMigrator rejects blank or repeated upsert_keys, upsert_keys with a write mode other than upsert, and upsert keys missing from the projected destination schema. An upsert with on_missing_in_source="skip" that would drop destination columns is rejected; with "null", the plan warns that matched keys get those values overwritten with null. SchemaReconciler.compute_plan reports a column_mapping key that is not a source column.

  • migrate-data --batch-rows and --sample-size must be at least 1, --upsert-keys ignores empty entries, and a missing destination table explains that migrate-data does not create it.

  • remove_orphan_files defaults to dry_run=True; pass dry_run=False to delete. It refuses tables with gc.enabled=false (in a dry run too), rejects a negative older_than_hours and sweep entries that are absolute or escape the table location, and warns when older_than_hours is below 24. deleted is True only when at least one file was removed.

  • All expire_snapshots arguments after table are keyword-only, and max_age_hours accepts fractional hours. Omitted arguments fall back to the table’s history.expire.max-snapshot-age-ms / history.expire.min-snapshots-to-keep properties (as written by set_snapshot_retention), then to the Iceberg defaults of 5 days and 1 snapshot. Invalid values raise ValueError. The fallback to PyIceberg’s internal ExpireSnapshots class for releases older than 0.11 was removed.

  • compact_small_files_streaming lost its unused batch_rows parameter.

Polaris (experimental):

  • Only system.snapshot-expiry is executed, and only with "enable": true (a JSON boolean). system.metadata-compaction no longer triggers a data-file rewrite; it, system.data-compaction and system.orphan-file-removal are reported as skipped.

  • system.snapshot-expiry reads the Polaris schema (config.max_snapshot_age_days, config.min_snapshot_to_keep; max_ref_age_days is ignored with a note). Legacy top-level *_ms keys, unknown top-level keys, unknown schema versions and invalid values fail the dispatch instead of running on defaults.

  • A custom handler returning None yields empty details; any other non-dict return marks the dispatch as failed.

  • run-policies is marked [experimental] and prints a warning when run, because its requests do not yet match the released Polaris policy API.

Fixed

Migrations and CLI:

  • Two concurrent migrate runs for the same namespace no longer leave duplicate history rows on PyIceberg 0.12 (apache/iceberg-python#4194). The run that records second raises ConcurrentMigrationError, and history keeps one row per version and namespace. Existing duplicate rows are read as their latest status and removed by the next write for that version.

  • When a migration’s own commit conflicts with a concurrent commit, it raises ConcurrentMigrationError and records nothing, instead of overwriting the other run’s history with failed.

  • migrate --target 2 no longer applies every migration, stamp --target 2 no longer stamps all of them, rollback --target 1 is no longer a no-op, and "2" no longer also selects 0010.

  • rollback(steps=0) and a negative steps raise ValueError instead of rolling back an unexpected number of migrations.

  • rollback checks that every migration file it needs exists before running any downgrade(). status lists history entries whose migration file is missing.

  • enforce_column is idempotent for struct, list and map columns.

  • enforce_column(..., legacy_archival=True) archives only on a real type incompatibility. Commit errors and failing contracts propagate instead of archiving the column and adding an empty replacement.

  • Archival, in enforce_column and in the enforce_rename collision path, refuses columns used by the partition spec, sort order or identifier fields. An archived required column becomes optional, so later appends no longer fail. If archival or the rename fails part way, the column is restored with its original doc and required flag; if that restore also fails, a RuntimeError is raised.

  • enforce_rename(name, name) is a no-op, and renaming across different parent structs raises ValueError.

  • enforce_column and add_column accept primitive PyArrow types such as pa.float64() and pa.string().

  • Migration files that define a @dataclass (with from __future__ import annotations) load. Same-named files in different namespace directories no longer collide. A broken migration file raises an error that names the file, and the CLI prints it as one Error: line.

  • create_table_if_not_exists works for nested namespaces such as sales.eu, and returns the existing table if another process creates it first.

  • set_table_properties accepts a property key named properties.

  • revision --autogenerate honours --catalog-name, the config’s catalog_name and the config-relative sqlite defaults.

  • Missing extras, missing tables, missing catalog properties, connection failures, permission or server errors, commit conflicts and corrupt checkpoints print a single Error: line and exit 1 instead of a traceback.

  • Config file problems (invalid YAML, a non-mapping top level, non-string values such as namespace: 123) are usage errors. Unknown keys print a warning.

  • Config and generated files are read and written as UTF-8 (a BOM is tolerated). init appends its block to .gitignore without rewriting the existing content, and honours --config and --migrations-dir.

  • Using the s3tables kind without boto3 says to install iceberg-ops[s3tables].

  • CatalogFactory.build(..., reset=True) refuses to delete a filesystem root, the home directory, the working directory or one of its parents, or a database file that is not SQLite.

  • The sqlite and memory kinds reject a warehouse that looks like a URI and local paths containing # or ?, which PyIceberg would misread, before anything is written.

  • The history table is created with commit.retry.num-retries=20, so runners for different namespaces sharing it no longer lose history writes to commit contention.

  • When recording a failure in history itself fails, the migration’s own error is still the one raised; the history error is logged.

  • history sizes its columns to their contents, so full ISO-8601 timestamps no longer push DESCRIPTION out of line, and a multi-line error is printed on one line.

  • create_table_if_not_exists raises ValueError before creating anything when a pa.Schema is combined with a non-empty partition_spec or sort_order.

  • SchemaContract.validate compares struct, list and map types without regard to field IDs, and enforce_rename checks the contract when neither column exists.

Autogenerate:

  • revision --autogenerate passes the table’s partition spec, sort order and format version to the diff. Dropping a column that the spec, the sort order or the identifier fields use makes the generated upgrade() raise before any step runs, and a type the table’s format version cannot hold is a usage error. Column names containing ., decimals with a scale outside 0..precision or more than 38 digits, and fixed[0] are rejected too. Existing nested columns of a --target-schema-arrow target that were not compared are named in a warning instead of reporting no differences.

  • Partition and sort fields sourced from a nested column render as the full dotted path and bind to the nested field.

  • Autogenerate no longer emits type changes that Iceberg does not allow, such as a decimal scale change, string to long, or binary to string; such a step makes upgrade() raise before anything runs.

  • A generated downgrade() with an irreversible step no longer rolls back part of the migration and then fails.

  • Nested type changes (struct, list, map) in a pyiceberg.schema.Schema target (Python API) produce a file that compiles, with the step blocked for manual completion, instead of a syntax error.

  • Invalid YAML contracts, unknown types and bad rename hints make revision --autogenerate exit with a usage error.

  • Partition fields with non-identity transforms (day, bucket, …) import their transform and run. Sort-order diffs compare transform and null order, and reject orders the generated call cannot express.

  • --target-schema-arrow modules are importable from the current directory when the CLI runs from the installed console script.

Data migration:

  • overwrite_partition could delete rows written by an earlier batch of the same migration; it is now limited to one commit (see Changed).

  • A resume reads the source snapshot recorded in the checkpoint, so rows added to the source after a crash are neither skipped nor duplicated. verify() after a run compares against the snapshot that was migrated.

  • Re-running with a checkpoint that records a completed migration raises CheckpointError instead of duplicating and dropping rows.

  • A batch committed just before a crash, but not yet checkpointed, is not written twice on resume.

  • Pre-flight runs every check before it changes the destination, so auto-added columns are no longer committed when a later check fails.

  • Writing to a destination partitioned by bucket, truncate, year, month, day or hour without pyiceberg-core fails before any write with an ImportError naming pip install 'iceberg-ops[transforms]'.

  • The first upsert after a schema-only change (for example an auto-added column) no longer fails; an empty snapshot is committed first, as a workaround for apache/iceberg-python#2467.

  • Composite-key upserts with more than a few thousand keys per batch no longer crash the interpreter (apache/iceberg-python#3508). Large batches are split into sub-batches of 1,000 keys within one commit.

  • When the source schema changed after the pinned snapshot, the plan uses the schema at that snapshot.

  • verify(method="partition_hash") works when a partition field’s name differs from its source column, and raises NotImplementedError for non-identity partitions instead of reporting a mismatch (the CLI prints Verify error: and exits 1).

  • A resume raises CheckpointError when the destination was rolled back past a checkpointed batch, instead of skipping that batch’s rows. Every refusal that follows committed batches names the snapshot to roll back to before reset().

  • Upsert keys that the plan would fill with null are rejected before any write.

  • DataMigrator and migrate-data refuse a source and destination that are the same table.

  • Checkpoint writes are fsync-ed before the atomic rename.

Maintenance:

  • remove_orphan_files no longer deletes live position-delete, equality-delete and deletion-vector files, or registered table and partition statistics files.

  • compact_small_files_streaming no longer deletes its staged files when the commit outcome is unknown (for example CommitStateUnknownException), which could make a committed table unreadable.

  • expire_snapshots skips branch and tag heads (including the current snapshot after a rollback) instead of aborting, and dry runs select the same snapshots as real runs. A max age older than the epoch expires nothing instead of overflowing.

  • Compaction and orphan removal accept Windows drive-letter table locations.

Security

  • Generated migration files cannot run injected code: the -m message and column, partition and property names and values are written as Python literals, and docstring text is escaped.

  • Polaris OAuth2 token-endpoint errors no longer echo the response payload, which could leak tokens to logs or stderr. The client strips surrounding whitespace from a token, rejects an empty token or one with inner whitespace or control characters (run-policies exits 2), and keeps the token and client secret out of connection-error messages.

  • A failed migration’s history row holds only "<Type>: <message>"; the traceback is logged locally at DEBUG level, since everyone with catalog access can read the history table.

  • remove_orphan_files matches file names after Unicode NFC normalisation and case folding, does not follow directory symlinks or Windows junctions, and never deletes a file whose real path is outside the table root.

  • Compaction escapes identity-partition values the way PyIceberg does and refuses to stage files outside <table>/data, so a partition value such as ../../x cannot write elsewhere.

  • run-policies reads ICEBERG_OPS_POLARIS_TOKEN, ICEBERG_OPS_POLARIS_CLIENT_ID and ICEBERG_OPS_POLARIS_CLIENT_SECRET (preferred over flags, which show up in process listings) and warns when --polaris-url uses plain http to a non-loopback host.

Known limitations

  • Concurrent runs for one namespace are not guarded, only detected: a second run can execute upgrade() before it fails to record history. Run one migration job per namespace at a time and keep upgrade() idempotent.

  • The history table keeps the latest status of each version per namespace, not one row per attempt. Earlier states are readable by time travel on the history table until its snapshots expire.

  • status, current, plan and history print only and exit 0 even when a migration is recorded as failed; there is no check command yet. The runner methods print their reports and return None.

  • CLI autogenerate diffs columns only. Autogenerate does not emit nullability or doc changes, blocks nested type changes from a pyiceberg.schema.Schema target, and does not compare existing nested columns of a pyarrow.Schema target (they are treated as unchanged; the CLI names them in a warning). Each generated step commits on its own, so a failing step leaves earlier steps applied.

  • Only SQLite and in-memory catalogs are integration-tested. Glue, REST, Hive and S3 Tables catalogs are reached through PyIceberg but have not been tested against live services.

  • Data migration: overwrite_partition is limited to one commit and identity-partitioned destinations; the destination table must already exist. batch_rows bounds the rows per commit, not peak memory, which grows with the number and size of source data files (apache/iceberg-python#2407); an upsert also loads the matched destination rows. Duplicate keys within one batch raise ValueError, and across batches the later row wins. verify="sample" can report a false mismatch on larger tables, and partition_hash needs an identity-partitioned or unpartitioned destination and reads both tables in full.

  • Maintenance: compaction and orphan removal support the local filesystem only. Compaction supports identity partitions and unpartitioned tables (other partitions are skipped), rewrites qualifying partitions whole, writes 2N + 1 snapshots for N partitions, uses PyArrow’s default codec, fails when a concurrent append lands in a partition being compacted, and its memory is not bounded by target_size_mb (apache/iceberg-python#2407). Delete files and deletion vectors are out of scope for compaction (there is no guard) and the health report. Snapshot expiry is metadata-only.

  • The Polaris executor does not call the Polaris 1.x policy endpoints, so fetching policies from a released Polaris server fails. It has not been validated against a live server.

  • Python 3.15 is not tested. At release time PyIceberg 0.12.0 published no Python 3.15 wheels.

  • PyArrow 26 and later refuse to import next to NumPy 1.x. iceberg-ops does not use NumPy; in an environment pinned to NumPy 1.x, also pin pyarrow<26.

  • On Windows, write a local warehouse in .pyiceberg.yaml as file://C:/path; PyArrow rejects the file:///C:/path form.