feat(python, rust): add statistics_enabled to ColumnProperties

[maxitg]

Description

• Add support for statistics_enabled in ColumnProperties.
• Remove deprecated and unused max_statistics_size from ColumnProperties.

Related Issue(s)

• [python] File size too large - maybe stats related #2965

Documentation

The only available way currently to avoid writing statistics to the transaction log for a column is via max_statistics_size. However, that has been deprecated in arrow-rs, furthermore it appears to be ignored: apache/arrow-rs#2033. Being able to skip statistics is necessary for large data columns, as otherwise transaction log explodes after a few tens of thousands of transactions and becomes non-scalable.

Examples

from deltalake import DeltaTable, write_deltalake, WriterProperties, ColumnProperties
import pandas as pd
import pyarrow as pa
from pathlib import Path
import shutil

data = {'int': [0, 1, 2], 'data': [b'0' * 100000, b'1' * 100000, b'2' * 100000]}

df = pd.DataFrame(data)

def check_size(data_column_properties: ColumnProperties):
    TMP_PATH = "/tmp/delta-test/delta-test-0"

    shutil.rmtree(TMP_PATH, ignore_errors=True)
    write_deltalake(
        TMP_PATH,
        df.sort_values(["int"]),
        schema=pa.schema([pa.field("int", pa.int64()), pa.field("data", pa.binary())]),
        mode="append",
        writer_properties=WriterProperties(
            compression="zstd",
            column_properties={"data": data_column_properties},
        ),
        engine="rust",
    )

    print(DeltaTable(TMP_PATH).to_pandas())

    return (Path(TMP_PATH) / "_delta_log/00000000000000000000.json").stat().st_size

[check_size(c)
 for c
 in [ColumnProperties(max_statistics_size=0), ColumnProperties(statistics_enabled="NONE")]]

#    int                                               data
# 0    0  b'00000000000000000000000000000000000000000000...
# 1    1  b'11111111111111111111111111111111111111111111...
# 2    2  b'22222222222222222222222222222222222222222222...
#    int                                               data
# 0    0  b'00000000000000000000000000000000000000000000...
# 1    1  b'11111111111111111111111111111111111111111111...
# 2    2  b'22222222222222222222222222222222222222222222...
# [201143, 1100]

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 72.24%. Comparing base (ef066c4) to head (c934459).

Files with missing lines	Patch %	Lines
python/src/lib.rs	0.00%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3126      +/-   ##
==========================================
- Coverage   72.28%   72.24%   -0.04%     
==========================================
  Files         134      134              
  Lines       42973    42988      +15     
  Branches    42973    42988      +15     
==========================================
- Hits        31062    31056       -6     
- Misses       9923     9940      +17     
- Partials     1988     1992       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ion-elgreco]

@maxitg looks good, could you remove the max_statistics_size as well?

[maxitg]

@maxitg looks good, could you remove the max_statistics_size as well?

@ion-elgreco I’ve added a second commit to remove it. I initially avoided it to prevent a breaking change.

[rtyler]

Thanks for the improvement! This is a situation where I'm definitely okay with breaking changes since the code wasn't working anyways, we're also on the precipice of a breaking set of changes anyways 😄