fix(core): resolve storage config aliases to canonical keys#540
Open
liwuhen wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
normalize_storage_configcurrently applies mirrored key pairs from a singlesnapshot of the normalized properties. This cannot resolve aliases that require
more than one mirror step.
For Azure, underscore aliases are declared through a dotted intermediate key:
azure.account_name→azure.account.nameazure.account.name→azure.account-nameHowever,
AzdlsConfigreads the hyphenated canonical keyazure.account-name. Because newly mirrored keys are not processed again,azure.account_nameis silently ignored. The same issue affects the otherdeclared Azure underscore aliases.
This change replaces symmetric mirroring with direct, ordered
alias → canonicalmappings. It also defines deterministic precedence whenmultiple forms are supplied:
For Azure, dotted aliases are declared before underscore aliases, resulting in:
canonical > dotted alias > underscore aliasBrief change log
normalize_storage_configfrom symmetric one-pass mirroring to directalias → canonicalresolution.hyphenated key read by
AzdlsConfig:account_nameaccount_keysas_tokenclient_idclient_secrettenant_idauthority_hostmapping contract.
Tests
test_azdls_config_parse_underscore_aliases(
storage_azdls.rs): data-driven coverage for every declared Azureunderscore alias and verification that each value reaches the corresponding
AzdlsConfigfield. Fails on the pre-fix code.test_azdls_config_alias_priority(storage_azdls.rs): verifiescanonical keys take precedence over aliases and dotted aliases take
precedence over underscore aliases.
does not regress S3, COS, OBS, or GCS configuration parsing.
CARGO_INCREMENTAL=0 cargo test -p paimon --features storage-azdls --lib storage_azdlsCARGO_INCREMENTAL=0 cargo test -p paimon --no-default-features --features storage-s3,storage-cos,storage-azdls,storage-obs,storage-gcs --lib io::storage_cargo fmt --all -- --checkgit diff --checkAPI and Format
No public API or persisted data format changes.
Supported external storage configuration aliases now resolve directly to the
canonical keys consumed by each backend. When multiple forms of the same
setting are supplied, precedence is deterministic: the canonical key wins,
followed by aliases in declaration order.
Documentation
No documentation changes required.