Migration History

iceberg_ops.history

MigrationHistory — stores migration records as a real Iceberg table inside a dedicated meta-namespace within the user’s own catalog.

Why Iceberg-backed (not SQLite)?

  • Works with any catalog backend — SQLite locally, Glue / Hive / REST in prod. The history moves with the data; no extra SQLite file to manage or replicate.

  • The history table benefits from all Iceberg guarantees: ACID writes, time travel, schema evolution.

  • A single history table tracks migrations for all managed namespaces via a namespace column, so one catalog deployment is all you need.

Default layout

_iceberg_ops/                  <- meta_namespace (configurable)
`-- migration_history          <- current status per (version, namespace)

Schema of migration_history

version      string  - zero-padded version string ("0001", ...)
namespace    string  - the user-managed namespace targeted
description  string  - human-readable migration summary
applied_at   string  - ISO-8601 UTC timestamp of last status change
status       string  - "applied" | "rolled_back" | "failed"
error        string  - error message excerpt when status="failed"
exception iceberg_ops.history.ConcurrentMigrationError[source]

Bases: RuntimeError

Raised when another process changed the migration history during a run.

Concurrent runs for one namespace are not prevented, only detected when history is written: the losing run raises this error and its status is not recorded. Run migrations for a namespace from one job at a time (for example a CI concurrency group) or put an external lock in front of the runner.

class iceberg_ops.history.MigrationHistory(catalog, meta_namespace='_iceberg_ops')[source]

Bases: object

Iceberg-backed migration state store.

The <meta_namespace> namespace and the migration_history table inside it are created on first write; reading an absent table returns empty results and creates nothing. No external database required.

Each (version, namespace) has one row holding its latest status; earlier states are not kept as rows. They remain readable via time travel on the history table’s snapshots until those snapshots expire.

Concurrent writers are not locked out. A write replaces the key’s row with a key-scoped overwrite; if another process committed a change to the same key after this handle last read the table, the write raises ConcurrentMigrationError instead of landing a duplicate row. A write that loses a race against writes for other keys (for example runners for other namespaces) is retried, up to the history table’s commit.retry.num-retries (20 on a table created by iceberg-ops).

Parameters:

  • catalog (Catalog) – The live PyIceberg catalog (same one the user manages data in).

  • meta_namespace (str, default: '_iceberg_ops') – Namespace dedicated to framework metadata. Users should never write their own tables here. Default: "_iceberg_ops".

SCHEMA = version: large_string not null namespace: large_string not null description: large_string applied_at: large_string not null status: large_string not null error: large_string
record_applied(version, namespace, description)[source]

Record a migration as successfully applied.

Parameters:

  • version (str) – Migration version ("0001").

  • namespace (str) – Target namespace of the migration.

  • description (str) – Human-readable migration summary.

Raises:

ConcurrentMigrationError – If another process changed this key’s row concurrently.

Return type:

None

record_rolled_back(version, namespace, description)[source]

Record a migration as rolled back.

Parameters:

  • version (str) – Migration version ("0001").

  • namespace (str) – Target namespace of the migration.

  • description (str) – Human-readable migration summary.

Raises:

ConcurrentMigrationError – If another process changed this key’s row concurrently.

Return type:

None

record_failed(version, namespace, description, error)[source]

Record a migration attempt that raised an exception.

Parameters:

  • version (str) – Migration version ("0001").

  • namespace (str) – Target namespace of the migration.

  • description (str) – Human-readable migration summary.

  • error (str) – Error text; truncated to 2000 characters.

Raises:

ConcurrentMigrationError – If another process changed this key’s row concurrently.

Return type:

None

get_applied_versions(namespace)[source]

Return the versions whose current status is "applied".

Parameters:

namespace (str) – Namespace to read history for.

Returns:

Applied versions. Empty when the history table does not exist yet (it is not created by reads).

Return type:

set[str]

get_all(namespace)[source]

Return the current history row of every version in namespace.

Parameters:

namespace (str) – Namespace to read history for.

Returns:

One row per version, ordered by version ascending. Each row has the keys version, namespace, description, applied_at, status and error. Empty when the history table does not exist yet.

Return type:

list[dict]