[core][spark] Support managed Format Table partition operations

[sundapeng]

Summary

Follow-up to #8707. This PR makes the catalog the source of truth for opt-in managed Format Table partitions and wires the complete partition lifecycle through Core and Spark SQL.

Changes

• Add catalog capability checks and REST-catalog partition gateways, while keeping existing unsupported configurations readable as unmanaged so operators can reset the option safely.
• Read managed partitions from paged catalog metadata with safe predicate-prefix pushdown. Registered partitions without directories read as empty, while unregistered filesystem data remains invisible until repaired.
• Register partitions after successful writes and preserve committed files when registration fails so metadata can be reconciled without data loss.
• Harden physical paths: validate untrusted partition metadata, keep resolved paths under the table root, preserve keyed-layout compatibility for . and .., and prevent managed table copies from switching partition-path layouts dynamically.
• Support managed Format Table ADD/DROP PARTITION, SHOW PARTITIONS, and MSCK REPAIR TABLE ... [ADD|DROP|SYNC] PARTITIONS in Spark. Partial DROP specs use bounded catalog paging and deduplicated scans.
• Make DROP failure states explicit: ambiguous catalog responses never trigger data deletion or registration recreation; directory-deletion failures remain unregistered so partial data is not exposed.
• Add sys.list_format_table_partitions with bounded filtered pagination. Continuation tokens are bound to the original where predicate and fail closed if a replayed raw catalog page changes.
• Add sys.sync_format_table_metadata with dry-run support, scoped filesystem/catalog reconciliation, raw partition-value preservation, paged catalog traversal, and bounded mutation batches.
• Document the managed option, Spark DDL, repair commands, procedures, limits, and migration path.

Testing

• Core managed scan, commit, compatibility, path-safety, and catalog tests (125 tests)
• FormatTableMetadataProcedureTest (36 tests)
• Spark DDL planning, partition management, and MSCK suites (59 tests)
• Spark 3 / Scala 2.13 compile on JDK 11
• Spark 4 / Scala 2.13 compile on JDK 17
• Spotless and Checkstyle for the affected Core and Spark modules
• GitHub Actions matrix

Notes

• Managed Format Table partitions remain opt-in through metastore.partitioned-table=true and require a catalog that advertises the capability (currently REST catalog).
• Filtered procedure page tokens are opaque and must be resumed with the same where expression.
• DROP errors identify whether catalog state is ambiguous or which unregistered paths need manual reconciliation.

Relates to #8707.

[sundapeng]

To make this easier to review, I have split it into three stacked PRs with identical content (the tip of the stack is byte-identical to this branch):

• [core][spark] Support catalog-managed Format Table reads and writes #8728 — [core][spark] catalog-managed reads and writes (engine-independent foundation, reviewable standalone)
• [spark] Support managed Format Table partition DDL #8729 — [spark] partition DDL (draft until [core][spark] Support catalog-managed Format Table reads and writes #8728 lands)
• [spark] Add managed Format Table partition administration #8730 — [spark] partition administration: procedures + MSCK (draft until [spark] Support managed Format Table partition DDL #8729 lands)

Converting this PR to draft and keeping it open as the full-feature view; review can start with #8728. I will close this one once the split PRs are merged.