Arrow / Iceberg Type Bridge

iceberg_ops.core.types

Bidirectional primitive type translation between PyArrow and PyIceberg, plus the Iceberg type-promotion rule used by autogenerate.

Public API

arrow_type_to_iceberg(t)

Map a PyArrow DataType to its Iceberg IcebergType counterpart. Covers booleans, signed integers, floats, strings (including large_string and string_view), binary (including large_binary and binary_view), fixed-size binary, the arrow.uuid extension type, dates, times, timestamps (with and without timezone), decimals, and dictionary-encoded columns (mapped by their value type). Nested types (list, struct, map) raise ValueError — callers that handle nested shapes must do so before delegating here.

iceberg_type_to_arrow(t)

Map an Iceberg type back to a PyArrow DataType. Routes through PyIceberg’s official schema_to_pyarrow so the returned type matches what PyIceberg readers and writers actually use; casts built against this result round-trip correctly.

is_allowed_promotion(old, new)

True only for the type changes the Iceberg spec allows on an existing column: int -> long, float -> double and decimal precision widening at the same scale.

Usage

from iceberg_ops.core.types import arrow_type_to_iceberg, iceberg_type_to_arrow

iceberg_t = arrow_type_to_iceberg(arrow_field.type)
arrow_t   = iceberg_type_to_arrow(iceberg_t)
iceberg_ops.core.types.arrow_type_to_iceberg(t)[source]

Translate a PyArrow DataType to an Iceberg IcebergType.

Parameters:

t (DataType) – The PyArrow type to translate.

Returns:

The matching Iceberg type. int8 / int16 / int32 collapse to IntegerType (32-bit); int64 becomes LongType. Timestamps with a timezone become TimestamptzType and naive timestamps become TimestampType. time32 / time64 become TimeType, fixed_size_binary(n) becomes FixedType(n), the arrow.uuid extension type becomes UUIDType, the string_view / binary_view types map like their plain counterparts, and a dictionary-encoded type maps to its value type.

Return type:

IcebergType

Raises:

ValueError – For types with no Iceberg primitive equivalent: nested types (lists, structs, maps — the caller must handle these), unsigned integers (Iceberg has none), decimals with more than 38 digits or a negative scale or one above the precision, and anything else unmapped. The message always starts with "Cannot map Arrow type".

Example

>>> arrow_type_to_iceberg(pa.int64())
LongType()
>>> arrow_type_to_iceberg(pa.timestamp("us", tz="UTC"))
TimestamptzType()
iceberg_ops.core.types.is_allowed_promotion(old, new)[source]

Return whether changing a column from old to new is an Iceberg type promotion.

The allow-list is the Iceberg v2 one: int -> long, float -> double, and decimal(p, s) -> decimal(q, s) with q > p (same scale). Every other change — narrowing, a scale change, string <-> binary, fixed[16] -> uuid, date -> timestamp, identical types — returns False.

PyIceberg’s own pyiceberg.schema.promote is deliberately not used: it accepts several of the changes above that the spec does not allow (for example a decimal scale change), and committing one can make existing data files unreadable.

Parameters:
Returns:

True when the change is an allowed promotion.

Return type:

bool

Example

>>> is_allowed_promotion(ice_types.IntegerType(), ice_types.LongType())
True
>>> is_allowed_promotion(ice_types.DecimalType(10, 2), ice_types.DecimalType(10, 4))
False
iceberg_ops.core.types.iceberg_type_to_arrow(t)[source]

Translate an Iceberg IcebergType to a PyArrow DataType.

Internally constructs a one-field sentinel schema and routes through PyIceberg’s pyiceberg.io.pyarrow.schema_to_pyarrow(), ensuring the returned type matches what PyIceberg’s own readers and writers use. Casts built against this result therefore round-trip correctly through the writer.

Parameters:

t (IcebergType) – The Iceberg type to translate.

Returns:

The matching PyArrow type.

Return type:

DataType

Example

>>> iceberg_type_to_arrow(ice_types.LongType())
DataType(int64)