Skip to content

fix(core): resolve storage config aliases to canonical keys#540

Open
liwuhen wants to merge 1 commit into
apache:mainfrom
liwuhen:fix/azure-underscore-aliases
Open

fix(core): resolve storage config aliases to canonical keys#540
liwuhen wants to merge 1 commit into
apache:mainfrom
liwuhen:fix/azure-underscore-aliases

Conversation

@liwuhen

@liwuhen liwuhen commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

normalize_storage_config currently applies mirrored key pairs from a single
snapshot of the normalized properties. This cannot resolve aliases that require
more than one mirror step.

For Azure, underscore aliases are declared through a dotted intermediate key:

  • azure.account_nameazure.account.name
  • azure.account.nameazure.account-name

However, AzdlsConfig reads the hyphenated canonical key
azure.account-name. Because newly mirrored keys are not processed again,
azure.account_name is silently ignored. The same issue affects the other
declared Azure underscore aliases.

This change replaces symmetric mirroring with direct, ordered
alias → canonical mappings. It also defines deterministic precedence when
multiple forms are supplied:

  1. An explicitly supplied canonical key
  2. The first matching alias in declaration order

For Azure, dotted aliases are declared before underscore aliases, resulting in:

canonical > dotted alias > underscore alias

Brief change log

  • Change normalize_storage_config from symmetric one-pass mirroring to direct
    alias → canonical resolution.
  • Preserve an explicitly supplied canonical value.
  • Resolve multiple aliases in declaration order.
  • Map every supported Azure dotted and underscore alias directly to the
    hyphenated key read by AzdlsConfig:
    • account_name
    • account_key
    • sas_token
    • client_id
    • client_secret
    • tenant_id
    • authority_host
  • Update the S3, COS, OBS, and GCS alias tables to follow the same directional
    mapping contract.

Tests

  • Unit test test_azdls_config_parse_underscore_aliases
    (storage_azdls.rs): data-driven coverage for every declared Azure
    underscore alias and verification that each value reaches the corresponding
    AzdlsConfig field. Fails on the pre-fix code.
  • Unit test test_azdls_config_alias_priority (storage_azdls.rs): verifies
    canonical keys take precedence over aliases and dotted aliases take
    precedence over underscore aliases.
  • Existing storage configuration tests verify the directional alias conversion
    does not regress S3, COS, OBS, or GCS configuration parsing.
  • Commands run locally:
    • CARGO_INCREMENTAL=0 cargo test -p paimon --features storage-azdls --lib storage_azdls
    • CARGO_INCREMENTAL=0 cargo test -p paimon --no-default-features --features storage-s3,storage-cos,storage-azdls,storage-obs,storage-gcs --lib io::storage_
    • cargo fmt --all -- --check
    • git diff --check

API and Format

No public API or persisted data format changes.

Supported external storage configuration aliases now resolve directly to the
canonical keys consumed by each backend. When multiple forms of the same
setting are supplied, precedence is deterministic: the canonical key wins,
followed by aliases in declaration order.

Documentation

No documentation changes required.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants