Polaris¶

Note

Experimental. The Apache Polaris policy executor has not been verified against a live Polaris server. PolarisClient does not yet call the Polaris 1.x policy endpoints, so fetching policies from a released Polaris server fails. Of the built-in policy types only system.snapshot-expiry runs, and only with "enable": true; the others are skipped unless you register a custom handler. The API may change in any release.

The iceberg_ops.polaris package re-exports its public surface from the three submodules below. Tests and downstream code can import the names from either the package (from iceberg_ops.polaris import PolarisClient) or the submodule – both bindings refer to the same object.

See Polaris Policy Executor (experimental) for the user-facing guide.

Package¶

iceberg_ops.polaris¶

Apache Polaris policy executor (experimental).

Polaris 1.0 added a policy framework: a way to declare table-level maintenance rules (snapshot expiry, compaction, orphan-file removal) at the catalog level via REST endpoints. Polaris itself does not execute the policies - an external worker has to fetch attached policies and invoke the matching maintenance operations against the underlying Iceberg tables.

This subpackage is a pure-Python, single-process prototype of that worker. It dispatches system.snapshot-expiry policies to iceberg_ops.maintenance.expire_snapshots(); every other policy type is skipped.

Experimental¶

Not verified against a live Apache Polaris server. In this release PolarisClient does not call the Polaris 1.x policy endpoints (/api/catalog/polaris/v1/{prefix}/...), so fetching policies from a real server fails with PolarisError. dispatch_policy() works on Policy objects you build yourself. The API may change in any release.

Public API¶

PolarisClient: Thin HTTP client for Polaris policy endpoints (paths unverified, see above). Lazy-imports requests, supports OAuth2 client-credentials auth or a pre-fetched static bearer token.
PolarisError: Raised on HTTP failures or unparseable responses.
Policy: Dataclass representing a parsed Polaris policy (name, policy_type, description, content, inheritable, version).
parse_policy(): Parse a raw policy dict into a Policy. content is a JSON string per the Polaris policy API; an already-decoded object is accepted too.
fetch_table_policies(): Convenience wrapper around PolarisClient.list_applicable_policies_for_table() that returns list[Policy] directly.
dispatch_policy(): Map one policy to its matching maintenance call against an Iceberg pyiceberg.table.Table. Only system.snapshot-expiry runs, and only with "enable": true. Returns a DispatchResult.
dispatch_policies(): Run dispatch_policy() over a list of policies and collect per-policy results. Continues past errors so one bad policy does not block the rest of the batch.
DispatchResult: Per-policy outcome (policy_name, policy_type, op, success, skipped, skip_reason, details, error).

CLI surface¶

The iceberg-ops run-policies CLI subcommand consumes this package - see iceberg_ops.cli.run_policies() for the entry point.

Lazy imports¶

The requests dependency is lazy-imported inside PolarisClient so installing the base package without [polaris] extras still imports iceberg_ops.polaris cleanly. Calls that need HTTP will raise a clear ImportError pointing at pip install 'iceberg-ops[polaris]'.

Test seam¶

Tests for downstream integrators can drive the client against iceberg_ops.testing.FakeSession instead of a live HTTP service - the client accepts any request / post -compatible session via the session= kwarg.

Client¶

iceberg_ops.polaris.client¶

Thin HTTP client for Apache Polaris policy endpoints.

Experimental¶

The endpoint paths this client builds have not been verified against a live Polaris server and do not match the Polaris 1.x policy API (/api/catalog/polaris/v1/{prefix}/...). Fetching policies from a real Polaris server therefore fails with PolarisError in this release. The client is kept so the policy pipeline can be driven end to end against iceberg_ops.testing.FakeSession.

Auth¶

The client supports two ways to provide auth:

token=<bearer> - a pre-fetched bearer token. Use this for external / OIDC identity providers: fetch the token yourself and pass it via token= (or --polaris-token on the CLI).
oauth2_credentials=(client_id, client_secret) - the client fetches a token with the client-credentials grant from base_url + token_endpoint (default /api/catalog/v1/oauth/tokens, Polaris’s internal token endpoint; still served by Polaris 1.8.0 but deprecated in the Iceberg REST spec) on first use.

Either way the token is cached for the lifetime of the client. There is no refresh on 401 or on expires_in. Polaris’s default token lifetime is one hour (PT1H), so create one client per run.

Lazy `requests` import¶

requests is imported inside the methods that hit the network so importing this module without the [polaris] extra installed does not blow up - only methods that actually make HTTP calls touch requests. Tests inject a session via the session= kwarg (typically iceberg_ops.testing.FakeSession) and skip the requests import entirely.

Endpoint paths¶

Policy requests go to base_url + management_prefix (default /api/management/v1) followed by /catalogs/{catalog}/namespaces/{ns}/.... These paths are unverified and differ from the Polaris 1.x policy API; see Experimental above.

exception iceberg_ops.polaris.client.PolarisError(message, *, status_code=None, body=None)[source]¶

Bases: Exception

Raised on HTTP failures or unparseable responses from Polaris.

The exception carries the HTTP status code (when one is available) and the response body so callers can surface useful diagnostics. status_code is None when the failure happened before a response was received (DNS failure, timeout, refused connection).

Variables:

status_code (int or None) – HTTP status code from the response, or None when no response was received.
body (str or None) – Truncated response body for diagnostics, or None when no response was received.

Parameters:

message (str)
status_code (int | None, default: None)
body (str | None, default: None)

class iceberg_ops.polaris.client.PolarisClient(base_url, catalog, *, token=None, oauth2_credentials=None, oauth2_scope='PRINCIPAL_ROLE:ALL', token_endpoint='/api/catalog/v1/oauth/tokens', management_prefix='/api/management/v1', timeout=30.0, session=None)[source]¶

Bases: object

Thin HTTP client for Polaris policy endpoints (experimental).

Experimental: the endpoint paths are unverified against a live Polaris server and do not match the Polaris 1.x policy API, so calls against a real server fail with PolarisError. The bearer token (static or OAuth2-fetched) is cached for the client’s lifetime with no refresh; create one client per run.

Parameters:

base_url (str) – Polaris host, e.g. "https://polaris.example.com". No trailing slash required; the client normalises it.
catalog (str) – Polaris catalog name. All policy endpoints are scoped to a catalog, so this is bound at construction time rather than per-call.
token (str | None, default: None) – Pre-fetched bearer token. Mutually exclusive with oauth2_credentials. Useful for tests and ad-hoc invocations. Surrounding whitespace (for example a trailing CR/LF read from a file) is stripped.
oauth2_credentials (tuple[str, str] | None, default: None) – (client_id, client_secret). The client fetches a token via OAuth2 client-credentials grant on first use.
oauth2_scope (str, default: 'PRINCIPAL_ROLE:ALL') – OAuth2 scope to request. Default "PRINCIPAL_ROLE:ALL" matches Polaris’s standard principal-role flow.
token_endpoint (str, default: '/api/catalog/v1/oauth/tokens') – Path on base_url to the OAuth2 token endpoint. Default "/api/catalog/v1/oauth/tokens" (Polaris internal auth).
management_prefix (str, default: '/api/management/v1') – Path prefix for the policy endpoints. Default "/api/management/v1" (unverified; see the module notes).
timeout (float, default: 30.0) – Per-request timeout in seconds. Default 30.
session (Any, default: None) – Inject a custom session for connection pooling or testing. Anything with the same request / post interface as requests.Session will work; tests use iceberg_ops.testing.FakeSession here.

Raises:

ValueError – If neither token nor oauth2_credentials is provided, or if both are, or if token is empty or contains whitespace or control characters after stripping. The message never includes the token.
ImportError – At first HTTP call if requests is not installed (i.e. the user installed the base package without [polaris]) and no session was supplied.

list_policies(namespace)[source]¶

List policies defined in namespace.

Returns the raw policy dicts as returned by the Polaris API. Use iceberg_ops.polaris.parse_policy() to turn each into a Policy if you want the parsed form.

Note: policies defined in a namespace may or may not be attached to specific tables. Use list_applicable_policies_for_table() to get the policies that actually apply to a given table (which includes inherited policies from parent namespaces).

Parameters:: namespace (str) – Namespace name. Multi-level namespaces are accepted in dot-separated form ("db.subdb") and encoded with Polaris’s %1F separator on the wire.
Returns:: Raw policy dicts as returned by Polaris.
Return type:: list[dict[str, Any]]

get_policy(namespace, name)[source]¶

Fetch a single policy by name from namespace.

Parameters:

namespace (str) – Namespace name (dot-separated for multi-level).
name (str) – Policy name within the namespace.

Returns:

Raw policy dict.

Return type:

dict[str, Any]

Raises:

PolarisError – On HTTP failure or when the response body is not a JSON object.

list_applicable_policies_for_table(namespace, table)[source]¶

List policies that apply to namespace.table.

Includes policies attached directly to the table and policies inherited from parent namespaces. The Polaris API returns the already-resolved set; the client does not re-walk the inheritance hierarchy itself.

Parameters:

namespace (str) – Namespace name (dot-separated for multi-level).
table (str) – Table name within the namespace.

Returns:

Raw policy dicts (the resolved applicable set).

Return type:

list[dict[str, Any]]

list_applicable_policies_for_namespace(namespace)[source]¶

List policies that apply at the namespace level.

Returns inherited + directly-attached policies for the namespace. Useful for users who want to drive maintenance from policies attached at namespace granularity rather than per-table.

Parameters:: namespace (str) – Namespace name (dot-separated for multi-level).
Returns:: Raw policy dicts (the resolved applicable set).
Return type:: list[dict[str, Any]]

Policies¶

iceberg_ops.polaris.policies¶

Policy dataclass + parsing helpers for the Polaris policy framework.

This module is the layer between the raw HTTP responses returned by PolarisClient and the dispatcher (iceberg_ops.polaris.dispatcher). It owns:

the Policy dataclass - a normalised view of a Polaris policy that downstream code can rely on;
parse_policy() - decodes content (a JSON string per the Polaris policy API; an already-decoded object is accepted too) and emits a Policy;
fetch_table_policies() - convenience wrapper that calls PolarisClient.list_applicable_policies_for_table() and parses the response into a list of Policy.

Experimental¶

Not verified against a live Apache Polaris server. The parser reads the policy type from a type key, and fetch_table_policies() relies on PolarisClient, whose endpoint paths do not match the Polaris 1.x policy API.

Why a dataclass and not just dicts?¶

The dispatcher reads policy.policy_type and policy.content against known schemas. Going through a dataclass:

gives a single place to decode content from its JSON-string form;
makes the contract typed at the dispatcher boundary so a malformed-policy bug fails at the parser, not deep inside the dispatcher;
lets tests construct fake Policy instances directly without touching HTTP at all.

class iceberg_ops.polaris.policies.Policy(name, policy_type, content, description=None, inheritable=False, version=0)[source]¶

Bases: object

A parsed Polaris policy (experimental).

Variables:

name (str) – Policy name within its namespace.
policy_type (str) – Polaris policy type identifier - e.g. "system.snapshot-expiry", "system.metadata-compaction". Custom (non-system.*) types are passed through verbatim; the dispatcher will skip anything it does not recognise. (Named policy_type rather than type so the attribute does not shadow the type builtin and so Sphinx’s cross-references stay unambiguous.)
content (dict) – Parsed policy parameters. Always a dict - parse_policy() decodes the JSON-string form the Polaris API uses.
description (str or None) – Free-text description configured on the policy.
inheritable (bool) – Whether the policy applies to nested namespaces / tables. Mostly informational at the executor level - Polaris itself resolves inheritance and returns the effective set via the applicable-policies endpoints.
version (int) – Polaris’s monotonic version counter for the policy.

Parameters:

name (str)
policy_type (str)
content (dict[str, Any])
description (str | None, default: None)
inheritable (bool, default: False)
version (int, default: 0)

name: str¶

policy_type: str¶

content: dict[str, Any]¶

description: str | None = None¶

inheritable: bool = False¶

version: int = 0¶

iceberg_ops.polaris.policies.fetch_table_policies(client, namespace, table)[source]¶

Fetch and parse all policies applicable to namespace.table.

Combines PolarisClient.list_applicable_policies_for_table() with parse_policy() so the caller gets a list of Policy ready to feed to the dispatcher.

Policies that fail to parse are dropped with a warning so a single malformed entry doesn’t block dispatch of the rest.

Parameters:

client (PolarisClient) – Client connected to the Polaris instance.
namespace (str) – Namespace name (dot-separated for multi-level).
table (str) – Table name within the namespace.

Returns:

Parsed policies, with any unparseable entries silently dropped (with a logged warning).

Return type:

list[Policy]

iceberg_ops.polaris.policies.parse_policy(raw)[source]¶

Parse a raw Polaris policy dict into a Policy.

The content field is handled as follows:

a string (the form the Polaris policy API uses) - parsed via json; it must decode to an object. An empty string yields an empty dict.
a dict - already decoded (for example a hand-built policy). Used as-is.
missing - an empty dict.

Anything else raises PolarisError so the caller has a single exception type to catch.

Parameters:: raw (dict[str, Any]) – One element of the list returned by list_policies() or one of the applicable-policies endpoints.
Returns:: Parsed policy.
Return type:: Policy
Raises:: PolarisError – On missing required fields (name, type) or unparseable content.

Dispatcher¶

iceberg_ops.polaris.dispatcher¶

Maps a Polaris Policy to the matching maintenance operation against an Iceberg pyiceberg.table.Table.

Experimental¶

The dispatcher has not been verified against a live Apache Polaris server. It works on Policy objects you build yourself (or parse with parse_policy()).

Supported policy types¶

One system.* policy type is executed:

system.snapshot-expiry -> iceberg_ops.maintenance.snapshots.expire_snapshots()

The other Polaris 1.x system types are recognised and always skipped:

system.metadata-compaction - manifest rewrite is not implemented.
system.data-compaction - compaction supports file:// tables only and is not executed from policies in this release.
system.orphan-file-removal - Polaris has defined this type since 1.0; orphan removal is never executed from policy automation in this release.

Every other type, including future system.* types, is skipped as unsupported unless a custom handler is registered for it. A skipped policy is returned as DispatchResult(skipped=True, op=None, skip_reason=...) so a batch run can still process the rest.

Enable gate¶

Every system.* policy that would run (built-in or overridden by a custom handler) runs only when content["enable"] is the boolean True. A missing enable, False, or a non-boolean value such as "true" or 1 skips the policy. Non-system.* custom types are not gated.

Snapshot expiry content¶

The content follows the Polaris system.snapshot-expiry schema (version 2025-02-03):

{
  "version": "2025-02-03",
  "enable": true,
  "config": {
    "max_snapshot_age_days": 2,
    "min_snapshot_to_keep": 1,
    "max_ref_age_days": 3
  }
}

Top-level keys other than version, enable and config are rejected (the dispatch fails rather than running on defaults). A version other than 2025-02-03 is rejected too.
config.max_snapshot_age_days -> max_age_hours (days x 24, fractional days allowed).
config.min_snapshot_to_keep -> min_snapshots (integer >= 1).
Both accept an int, a float, or a numeric string; booleans, NaN, infinities and negative values are rejected.
config.max_ref_age_days is ignored with a note: branch and tag heads are never expired.
Other config keys are ignored with a note.
Missing keys are not passed, so expire_snapshots falls back to the table’s history.expire.* properties and then its defaults.

Custom policy types¶

Polaris cannot store custom policy types, so handlers= is for policy sources other than Polaris, or for overriding the built-in behaviour of a type:

def handle_my_policy(table, policy, *, dry_run):
    ...
    return {"k": "v"}  # included in DispatchResult.details

dispatch_policy(
    table,
    policy,
    handlers={"my.org.weird-policy": handle_my_policy},
)

The handler receives (table, policy, *, dry_run) and returns a dict of details (None is treated as {}; any other non-dict return marks the dispatch as failed) or raises on failure.

Dry-run¶

dispatch_policy(..., dry_run=True) forwards dry_run=True to the maintenance op. The result reports what would happen without mutating the table.

class iceberg_ops.polaris.dispatcher.DispatchResult(policy_name, policy_type, op=None, success=False, skipped=False, skip_reason=None, details=dict(), error=None)[source]¶

Bases: object

Outcome of dispatching one Policy against a table.

Variables:

policy_name (str) – policy.name of the policy that was dispatched.
policy_type (str) – policy.policy_type of the policy that was dispatched.
op (str or None) – Name of the maintenance op that was invoked, or None if the dispatch was skipped before any op ran.
success (bool) – True when the op ran (or would have run, in dry-run) to completion. False when the op raised or a handler returned a non-dict. Always False when skipped is True.
skipped (bool) – True when the dispatcher chose not to run anything: the policy type is unsupported, or a system.* policy is not enabled with "enable": true.
skip_reason (str or None) – Human-readable reason populated alongside skipped=True.
details (dict) – Per-op metadata. Includes the parsed kwargs, the maintenance helper’s result fields, and any notes about ignored options.
error (str or None) – Stringified exception when success=False due to a raised exception or an invalid handler return. None for skips.

Parameters:

policy_name (str)
policy_type (str)
op (str | None, default: None)
success (bool, default: False)
skipped (bool, default: False)
skip_reason (str | None, default: None)
details (dict[str, Any], default: dict())
error (str | None, default: None)

policy_name: str¶

policy_type: str¶

op: str | None = None¶

success: bool = False¶

skipped: bool = False¶

skip_reason: str | None = None¶

details: dict[str, Any]¶

error: str | None = None¶

iceberg_ops.polaris.dispatcher.dispatch_policy(table, policy, *, dry_run=False, handlers=None)[source]¶

Dispatch one Policy against table.

Returns a DispatchResult describing the outcome. Exceptions raised by the underlying maintenance op are caught and surfaced via DispatchResult.error so a batch run can keep going past a single broken policy.

Parameters:

table (Table) – Target Iceberg table.
policy (Policy) – Parsed policy (from parse_policy() or fetch_table_policies()).
dry_run (bool, default: False) – When True, forward dry_run=True to the underlying maintenance op so no mutations occur.
handlers (dict[str, Callable[..., dict[str, Any] | None]] | None, default: None) – Map of policy_type -> handler(table, policy, *, dry_run). Keys take precedence over built-in handling, including the always-skipped system types. A handler for a system.* type still runs only when the policy has "enable": true.

Returns:

Outcome of the dispatch.

Return type:

DispatchResult

iceberg_ops.polaris.dispatcher.dispatch_policies(table, policies, *, dry_run=False, handlers=None)[source]¶

Dispatch a batch of policies, returning one result per policy.

Continues past errors so one bad policy does not block the rest. Order in the returned list matches input order.

Parameters:

table (Table) – Target Iceberg table.
policies (list[Policy]) – Parsed policies to dispatch in order.
dry_run (bool, default: False) – When True, forward dry_run=True to each maintenance op.
handlers (dict[str, Callable[..., dict[str, Any] | None]] | None, default: None) – Map of policy_type -> handler for custom policy types.

Returns:

One per input policy, same order.

Return type:

list[DispatchResult]

Polaris¶

Package¶

iceberg_ops.polaris¶

Experimental¶

Public API¶

CLI surface¶

Lazy imports¶

Test seam¶

Client¶

iceberg_ops.polaris.client¶

Experimental¶

Auth¶

Lazy requests import¶

Endpoint paths¶

Policies¶

iceberg_ops.polaris.policies¶

Experimental¶

Why a dataclass and not just dicts?¶

Dispatcher¶

iceberg_ops.polaris.dispatcher¶

Experimental¶

Supported policy types¶

Enable gate¶

Snapshot expiry content¶

Custom policy types¶

Dry-run¶

Lazy `requests` import¶