Maintenance

Note

Basic, frozen in scope. These are basic helpers for self-managed tables on the local filesystem: compaction and orphan cleanup support local (file://) table locations only, and delete files and v3 deletion vectors are out of scope. The helpers receive bug fixes only, and, like the rest of the Alpha API, may change in any release before 1.0. See Storage Maintenance for the known limitations.

Storage maintenance operations for Iceberg tables. Only needed when your catalog does not provide managed maintenance. The package re-exports its public functions and result dataclasses from the four submodules below; tests and downstream code can import the names from either the package (from iceberg_ops.maintenance import compact_small_files_streaming) or the submodule – both bindings refer to the same object.

Package

iceberg_ops.maintenance

Storage maintenance for Apache Iceberg tables.

These are basic helpers for self-managed tables on the local filesystem. Delete files and v3 deletion vectors are out of scope. They are only needed when your catalog does not provide managed maintenance; AWS S3 Tables, Snowflake-managed Iceberg, and Databricks Unity Catalog handle compaction / snapshot expiry / orphan cleanup automatically.

For self-hosted deployments, schedule these helpers on a cadence appropriate for your write rate (e.g. cron, or a runner like the one in iceberg_ops.runner).

Public API

compact_small_files_streaming(table, *, target_size_mb=512, ...)()

Partition-aware compaction for Identity-partitioned (or unpartitioned) tables. Rewrites qualifying partitions into Parquet files of roughly target_size_mb and swaps them in via one PyIceberg transaction. Peak memory grows with the files per partition; it is not bounded by target_size_mb.

expire_snapshots(table, *, max_age_hours=None, min_snapshots=None, dry_run=False)()

Expire snapshots older than max_age_hours while keeping at least min_snapshots and every branch / tag head. Unset arguments fall back to the table’s history.expire.* properties. Metadata-only.

remove_orphan_files(table, *, older_than_hours=72, sweep=("data", "metadata"), dry_run=True)()

Recursively walk the table location and report (or, with dry_run=False, delete) files that no table metadata references and whose mtime predates the cutoff. Refuses tables with gc.enabled=false.

table_health_report(table, small_file_threshold_mb=100)()

Read-only diagnostic of file layout, snapshot count, and partition surface, with actionable recommendations. Reads inspect.files(), which builds one Python dict per file, so cost grows with the file count.

Local filesystem only

The compaction and orphan-cleanup helpers support file:// locations only. Tables on s3:// / gs:// / abfs:// locations raise NotImplementedError with a clear message. expire_snapshots() and table_health_report() only touch metadata and work on every backend.

The APIs may change before 1.0.

Compaction

iceberg_ops.maintenance.compaction

Rewrite small data files into larger ones to improve query performance.

Public API

compact_small_files_streaming(table, *, target_size_mb=512, ...)

Partition-aware compaction for Identity-partitioned (or unpartitioned) tables. Reads each qualifying partition via Table.scan().to_arrow_batch_reader(), writes pre-staged Parquet files of roughly target_size_mb, then swaps them in with a single PyIceberg transaction.

Memory

PyIceberg’s batch reader decodes whole files before yielding batches (apache/iceberg-python#2407, apache/iceberg-python#4028), so peak memory grows with the number and size of files per partition. It is not bounded by target_size_mb.

Scope

A basic helper for self-managed tables on the local filesystem (file:// URIs and plain paths, including Windows drive-letter paths; other schemes raise NotImplementedError). Delete files and v3 deletion vectors are out of scope.

class iceberg_ops.maintenance.compaction.CompactionResult(files_before, files_after, bytes_before, bytes_after, files_rewritten, skipped=False)

Bases: object

Result of a compaction operation.

Variables:

• files_before (int) – Total data files before compaction.

• files_after (int) – Total data files after compaction.

• bytes_before (int) – Total bytes across all data files before compaction.

• bytes_after (int) – Total bytes across all data files after compaction.

• files_rewritten (int) – Number of small files that were (or would be) rewritten.

• skipped (bool) – True if compaction was skipped (no partition met the min_file_count threshold).

Parameters:

• files_before (int)

• files_after (int)

• bytes_before (int)

• bytes_after (int)

• files_rewritten (int)

• skipped (bool, default: False)

files_before: int

files_after: int

bytes_before: int

bytes_after: int

files_rewritten: int

skipped: bool = False

iceberg_ops.maintenance.compaction.compact_small_files_streaming(table, *, target_size_mb=512, min_file_count=5, max_table_size_bytes=None, sort_by_iceberg_order=True, dry_run=False)

Partition-aware compaction of small data files.

Reads each qualifying partition through a PyArrow RecordBatchReader, writes pre-staged Parquet files of roughly target_size_mb, then swaps them in with a single PyIceberg transaction.

Algorithm

1. Read inspect.files() to collect, per partition, the list of small files (size < target_size_mb) and total bytes.

2. For each partition with at least min_file_count small files:

   1. Open table.scan(row_filter=<partition predicate>) .to_arrow_batch_reader().

   2. Buffer batches until buffer.nbytes >= target_size_mb.

   3. (Optional) Pre-sort the buffer via table.sort_order() if sort_by_iceberg_order=True and the table has a sort order.

   4. Flush each full buffer to a NEW Parquet file at <table.location>/data/<partition_dir>/_iceberg_ops_compacted_<uuid>_<seq>.parquet. <partition_dir> escapes names and values the way PyIceberg does (region=a%2Fb), so every staged file stays under <table.location>/data.

3. Swap - build one PyIceberg transaction: for each touched partition, tx.overwrite(empty_df, overwrite_filter=<partition predicate>) removes the old files; tx.add_files(<staged URIs>) registers the new ones; then commit. The commit writes 2N + 1 snapshots for N touched partitions (a delete and an empty append per partition, then one append), which incremental readers see as new data.

Failure outcomes

• Staging or transaction building fails: nothing was committed; the staged files are deleted and the error is re-raised.

• The catalog rejects the commit (CommitFailedException or ValidationException): nothing was committed; the staged files are deleted and the error is re-raised.

• Any other commit error (for example CommitStateUnknownException or a timeout): the commit may have landed, so the staged files are kept, an error is logged, and the error is re-raised. Refresh the table and verify; if the commit did not land, remove_orphan_files() reclaims the files once they are older than its older_than_hours.

param table:

The Iceberg table to compact.

type table:

Table

param target_size_mb:

Files smaller than this are considered “small”. Each compacted Parquet buffer also caps at this size in memory. Default 512.

type target_size_mb:

int, default: 512

param min_file_count:

A partition is only compacted when it has at least this many small files. Default 5.

type min_file_count:

int, default: 5

param max_table_size_bytes:

Refuse to run when the total data-file size exceeds this value. Default None (no limit). Useful as a guard in scheduled jobs that touch unfamiliar tables.

type max_table_size_bytes:

int | None, default: None

param sort_by_iceberg_order:

If True (default), pre-sort each output buffer by the table’s sort order before writing - matches the behaviour S3 Tables and Snowflake apply automatically.

type sort_by_iceberg_order:

bool, default: True

param dry_run:

If True, report what would happen without writing or committing. Default False.

type dry_run:

bool, default: False

returns:

Statistics about the compaction.

rtype:

CompactionResult

raises ValueError:

If target_size_mb is not positive, or a staging directory would resolve outside <table.location>/data (staged files are removed).

raises RuntimeError:

If max_table_size_bytes is set and exceeded.

raises NotImplementedError:

If the table’s location uses a non-local scheme (s3:// etc). Plain paths, including Windows drive-letter paths, are local.

raises pyiceberg.exceptions.CommitFailedException:

If the catalog rejects the commit (staged files are removed).

raises pyiceberg.exceptions.ValidationException:

If a concurrent commit added files to a partition being compacted (staged files are removed).

Notes

Only Identity-transformed partition fields are supported. Partitions using Bucket / Truncate / Year / Month / Day / Hour transforms are skipped (with a warning log) because their partition struct values cannot be inverted to a row predicate without scanning all data. Qualifying partitions are rewritten whole, and output files use PyArrow’s default codec rather than the table’s configured codec. A concurrent append to a partition being compacted fails the run with ValidationException and the staged files are removed. PyIceberg retries the commit after appends to other partitions; those appends are preserved and counted in files_after. Delete files and v3 deletion vectors are out of scope.

Peak memory is not bounded by target_size_mb: PyIceberg’s batch reader decodes whole source files (apache/iceberg-python#2407), so memory grows with the number and size of files per partition.

Snapshot Expiry

iceberg_ops.maintenance.snapshots

Expire old snapshots based on age and minimum count policy.

Public API

expire_snapshots(table, *, max_age_hours=None, min_snapshots=None, dry_run=False)

Identify and expire snapshots older than max_age_hours, while always keeping at least min_snapshots and never expiring a branch or tag head. Unset arguments fall back to the table’s history.expire.* properties, then to the Iceberg defaults (5 days, 1 snapshot).

Expiry is metadata-only: the snapshots are removed from table metadata but their data files stay on storage until an orphan cleanup (see iceberg_ops.maintenance.remove_orphan_files(), file:// only) reclaims them.

class iceberg_ops.maintenance.snapshots.SnapshotExpiryResult(total_snapshots, expired_count, retained_count, expired_ids=list(), protected_ids=list())

Bases: object

Result of a snapshot expiry operation.

Variables:

• total_snapshots (int) – Number of snapshots before the operation.

• expired_count (int) – Number of snapshots expired (or identified for expiry in dry-run).

• retained_count (int) – Number of snapshots retained after the operation.

• expired_ids (list[int]) – Snapshot IDs that were (or would be) expired.

• protected_ids (list[int]) – Snapshot IDs that were old enough to expire but were kept because a branch or tag points at them.

Parameters:

• total_snapshots (int)

• expired_count (int)

• retained_count (int)

• expired_ids (list[int], default: list())

• protected_ids (list[int], default: list())

total_snapshots: int

expired_count: int

retained_count: int

expired_ids: list[int]

protected_ids: list[int]

iceberg_ops.maintenance.snapshots.expire_snapshots(table, *, max_age_hours=None, min_snapshots=None, dry_run=False)

Expire old snapshots based on age and minimum count policy.

Snapshots are sorted newest-first. The function expires snapshots older than max_age_hours but always retains the min_snapshots most recent ones. Branch and tag heads (including the current snapshot after a rollback) are never expired; they count towards min_snapshots. Dry-run and real runs select the same snapshots.

Expiry is metadata-only: data files of expired snapshots stay on storage until an orphan cleanup reclaims them.

Parameters:

• table (Table) – The Iceberg table to manage.

• max_age_hours (float | None, default: None) – Expire snapshots older than this many hours. When None (the default), the table property history.expire.max-snapshot-age-ms is used, falling back to the Iceberg default of 5 days (120 hours). An age that reaches back before the Unix epoch (for example the property set to 9223372036854775807, Java’s Long.MAX_VALUE) expires nothing by age.

• min_snapshots (int | None, default: None) – Always keep at least this many snapshots. When None (the default), the table property history.expire.min-snapshots-to-keep is used, falling back to the Iceberg default of 1.

• dry_run (bool, default: False) – If True, report which snapshots would be expired without acting. Default False.

Returns:

Statistics about expired snapshots.

Return type:

SnapshotExpiryResult

Raises:

ValueError – If the resolved min_snapshots is below 1 or the resolved max_age_hours is negative, or if a history.expire.* property is not an integer.

Notes

PyIceberg commits the expiry without asserting the table’s refs, so a tag created on an expiring snapshot between this function’s refresh and its commit is dropped with the snapshot. Avoid creating tags while an expiry runs.

Orphan Cleanup

iceberg_ops.maintenance.orphans

Remove storage files not referenced by any current Iceberg metadata.

Public API

remove_orphan_files(table, *, older_than_hours=72, sweep=("data", "metadata"), dry_run=True)

Build the set of file names the table’s metadata references, then walk the sweep directories on the local filesystem and report (or, with dry_run=False, delete) any file whose basename is not referenced AND whose mtime is older than older_than_hours.

The referenced set covers every file reachable from table metadata: manifest lists, manifests, and every live manifest entry of every snapshot (data files, position-delete files, equality-delete files and deletion-vector Puffin files), plus table statistics files, partition statistics files, every metadata JSON in the metadata log and the current metadata JSON.

Safety rails

• dry_run defaults to True; pass dry_run=False to delete.

• Tables with gc.enabled=false are refused, in dry-run too.

• Sweep directories must stay inside the table location. The walk does not follow directory symlinks or Windows junctions, and a file whose real path lies outside the table location is never a candidate.

• Names are matched case-insensitively after Unicode NFC normalisation, so a file whose metadata spelling differs only in case or normalisation form from its on-disk name is kept.

• Never run this against a location shared with another table: files the other table references would look like orphans here.

Local filesystem only

This implementation supports local locations only: file:// URIs and plain paths, including Windows drive-letter paths. Tables on s3:// / gs:// / abfs:// raise NotImplementedError.

class iceberg_ops.maintenance.orphans.OrphanCleanupResult(referenced_count, orphan_count, orphan_bytes, orphan_paths=list(), deleted=False, failed_paths=list())

Bases: object

Result of an orphan file cleanup operation.

Variables:

• referenced_count (int) – Number of distinct file basenames referenced by table metadata (manifest lists, manifests, data and delete files, statistics files, and metadata JSON files), counted after case folding and Unicode NFC normalisation.

• orphan_count (int) – Number of orphan files found. In a non-dry run this counts every candidate, including any listed in failed_paths.

• orphan_bytes (int) – Total bytes of orphan files found.

• orphan_paths (list[str]) – Local paths of orphan files found, in directory walk order.

• deleted (bool) – True if at least one file was actually deleted (never True in a dry run).

• failed_paths (list[str]) – Orphan paths whose deletion raised OSError. Always empty in a dry run.

Parameters:

• referenced_count (int)

• orphan_count (int)

• orphan_bytes (int)

• orphan_paths (list[str], default: list())

• deleted (bool, default: False)

• failed_paths (list[str], default: list())

referenced_count: int

orphan_count: int

orphan_bytes: int

orphan_paths: list[str]

deleted: bool = False

failed_paths: list[str]

iceberg_ops.maintenance.orphans.remove_orphan_files(table, *, older_than_hours=72, sweep=('data', 'metadata'), dry_run=True)

Find, and optionally delete, files no longer referenced by the table.

The referenced set is every manifest list, every manifest, every live manifest entry (data files, position-delete files, equality-delete files and deletion-vector Puffin files) of every snapshot, every table and partition statistics file, every metadata JSON in the metadata log, and the current metadata JSON. Each sweep directory is then walked recursively; a file is treated as an orphan iff its basename is not in the referenced set, its extension matches a known Iceberg suffix (.parquet / .avro / .metadata.json / .puffin / .stats), and its mtime predates the cutoff.

Matching is by basename and is conservative: a file is kept if any referenced path shares its name, even in another directory. Names are compared case-insensitively after Unicode NFC normalisation, because Windows and default macOS filesystems resolve them that way. The walk does not follow directory symlinks or Windows junctions, and a file whose real path lies outside the table location is never a candidate.

Never run this against a location shared with another table. Files the other table references would look like orphans here.

Parameters:

• table (Table) – The Iceberg table to clean.

• older_than_hours (int, default: 72) – Files newer than this are skipped. Default 72. Values below 24 log a warning because files of in-flight writers may be deleted.

• sweep (tuple[str, ...], default: ('data', 'metadata')) – Subdirectories of table.location() to walk. Default ("data", "metadata"). Entries must be relative and stay inside the table location.

• dry_run (bool, default: True) – If True (the default), report orphans without deleting. Pass dry_run=False to delete them.

Returns:

Statistics about referenced and orphan files. deleted is True only when at least one file was actually removed; failed_paths lists orphans whose deletion failed.

Return type:

OrphanCleanupResult

Raises:

• NotImplementedError – If table.location() uses a non-local scheme (e.g. s3://). Plain paths, including Windows drive-letter paths, are local.

• ValueError – If older_than_hours is negative, a sweep entry is absolute or escapes the table location, or the table has gc.enabled=false (raised in dry-run too).

Health Report

iceberg_ops.maintenance.advisor

Read-only diagnostic of table file layout and health.

Public API

table_health_report(table, small_file_threshold_mb=100)

Generate a health report with file statistics and recommendations. Never modifies the table - safe to run from any context.

class iceberg_ops.maintenance.advisor.TableHealthReport(total_files, total_size_bytes, avg_file_size_bytes, min_file_size_bytes, max_file_size_bytes, small_file_count, small_file_percentage, snapshot_count, oldest_snapshot_age_hours, partition_field_count, recommendations=list())

Bases: object

Read-only diagnostic of table file layout and health.

Variables:

• total_files (int) – Number of data files in the current snapshot.

• total_size_bytes (int) – Sum of all data file sizes.

• avg_file_size_bytes (float) – Average file size in bytes.

• min_file_size_bytes (int) – Smallest data file size.

• max_file_size_bytes (int) – Largest data file size.

• small_file_count (int) – Number of files below small_file_threshold_mb.

• small_file_percentage (float) – Percentage of files that are “small”.

• snapshot_count (int) – Total number of snapshots in metadata.

• oldest_snapshot_age_hours (float) – Age of the oldest snapshot in hours.

• partition_field_count (int) – Number of active partition fields in the current spec.

• recommendations (list[str]) – Human-readable maintenance suggestions.

Parameters:

• total_files (int)

• total_size_bytes (int)

• avg_file_size_bytes (float)

• min_file_size_bytes (int)

• max_file_size_bytes (int)

• small_file_count (int)

• small_file_percentage (float)

• snapshot_count (int)

• oldest_snapshot_age_hours (float)

• partition_field_count (int)

• recommendations (list[str], default: list())

total_files: int

total_size_bytes: int

avg_file_size_bytes: float

min_file_size_bytes: int

max_file_size_bytes: int

small_file_count: int

small_file_percentage: float

snapshot_count: int

oldest_snapshot_age_hours: float

partition_field_count: int

recommendations: list[str]

iceberg_ops.maintenance.advisor.table_health_report(table, small_file_threshold_mb=100)

Generate a read-only health report for an Iceberg table.

Aggregates file sizes with pyarrow.compute. The input comes from table.inspect.files(), which builds one Python dict per file, so time and memory grow with the number of files. Also computes snapshot count and oldest-snapshot age for retention recommendations.

This function never modifies the table. Safe to run from any context including read-only Lambda functions.

Parameters:

• table (Table) – The Iceberg table to inspect.

• small_file_threshold_mb (int, default: 100) – Files smaller than this are counted as “small”. Default 100.

Returns:

Diagnostic data with actionable recommendations.

Return type:

TableHealthReport