[datafusion] Support Utf8View schemas and residual filtering

[JingsongLi]

What changed

• expose Paimon CHAR / VARCHAR columns to DataFusion as Utf8View
• strip storage-only Parquet field-id metadata from the DataFusion-facing schema
• cast reader batches to the declared DataFusion schema while preserving zero-column row counts
• normalize string scalars for equality, ordering, membership, range, and pattern residual filters
• update MERGE, UPDATE, SQL context, table-provider, Parquet, and TPC-DS regression tests

Why

The TPC-DS benchmark uncovered a mismatch between the schema advertised to DataFusion and the physical string arrays returned by Paimon scans. String residual predicates also constructed Utf8 scalars even when the decoded column used Utf8View, causing Arrow comparison kernels to reject otherwise valid predicates.

This PR keeps the compatibility change self-contained. Runtime-filter, pruning, statistics, and reader batch-size work remain in later PRs.

Validation

• cargo check -p paimon -p paimon-datafusion -p paimon-tpcds-bench
• cargo test -p paimon test_evaluate_string_view_comparison_families
• cargo test -p paimon-datafusion --lib test_datafusion_schema_hides_paimon_field_ids
• cargo test -p paimon-datafusion --test pk_tables test_pk_aggregation_count_star_empty_projection -- --exact
• cargo test -p paimon-tpcds-bench
• cargo clippy -p paimon -p paimon-datafusion -p paimon-tpcds-bench --all-targets -- -D warnings
• cargo fmt --all -- --check

A local full paimon-datafusion --lib run passed 278 tests; seven existing fixture-dependent tests reported TableNotExist because the external /tmp/paimon-warehouse tables were not populated.

[leaves12138]

I found two blockers in the current revision (94d76ed):

1. Utf8View leaks through the Python API and breaks common PyArrow operations.

   datafusion_arrow_schema now advertises Paimon string columns as Utf8View, and bindings/python/src/context.rs exports the resulting batches to PyArrow without conversion. With the CI version of PyArrow (23.0.1), both write round-trip tests fail because Table.sort_by() uses take, for which PyArrow has no string_view kernel:

   ArrowNotImplementedError: Function 'array_take' has no kernel matching input types (string_view, uint64)
   

   This is not only a test expectation issue: existing Python consumers can receive batches on which common PyArrow operations fail. Please preserve Utf8 at the Python boundary (or provide another explicit compatibility strategy). Merely changing the tests to avoid sort_by() would leave the user-facing regression in place.

2. The DataFusion integration suite was not migrated to the new string array type.

   The integration (datafusion) job currently reports 25 failures in append_merge_into. Helpers such as collect_int_int_str and collect_int_str in crates/integrations/datafusion/tests/common/mod.rs still unconditionally downcast query results to StringArray, so they panic after the schema change. There are additional direct StringArray downcasts in pk_tables.rs, read_tables.rs, and other integration tests.

   Please update the full integration suite. A shared helper that reads Utf8, LargeUtf8, and Utf8View would be preferable to coupling every test to one physical representation.

The focused residual-filter, DataFusion schema, zero-column projection, and TPC-DS tests pass locally; I did not find another wrong-read in those implementation paths.

[leaves12138]

There is still one Python compatibility path that is not covered by the current fix (7ef01dc0).

pyarrow_compatible_batch only runs for the bundled pypaimon_rust.datafusion.SQLContext.sql() method. PaimonCatalog is also a public Python API intended to be registered into DataFusion's native Python SessionContext, and that path bypasses this conversion entirely.

I reproduced it locally against this head with PyArrow 23.0.1 and DataFusion 54.0.0:

internal = SQLContext()
internal.register_catalog("paimon", {"warehouse": warehouse})
internal.sql("CREATE TABLE paimon.default.t (id INT, name STRING)")
internal.sql("INSERT INTO paimon.default.t VALUES (2, 'b'), (1, 'a')")

external = datafusion.SessionContext()
external.register_catalog_provider(
    "paimon", PaimonCatalog({"warehouse": warehouse})
)
table = pa.Table.from_batches(
    external.sql("SELECT id, name FROM paimon.default.t").collect()
)

The bundled SQLContext now returns name: string and sort_by succeeds, but the external SessionContext path still returns:

id: int32
name: string_view

and table.sort_by("id") fails with:

ArrowNotImplementedError: Function 'array_take' has no kernel matching input types (string_view, uint64)

Please also preserve compatible Utf8 batches for the PaimonCatalog FFI provider path, or explicitly define why the two public Python query APIs should expose different string types. A regression test using SessionContext.register_catalog_provider plus sort_by would cover this path.

[leaves12138]

Re-reviewed the latest revision. The external Python catalog-provider path now preserves Utf8 schemas, and the new regression test covers SessionContext.register_catalog_provider together with PyArrow sort_by. I also verified the String primary-key merge test locally. All CI checks pass. LGTM.