Crash during justification import at epoch boundary leaves ledger arena permanently corrupted (GRANDPA qed-panic → unrecoverable 'root should be in the arena' crash loop)

[scottbuckel]

Summary

On a Preprod bootnode running 1.0.0-8af7d08a, a GRANDPA justification-import panic at an epoch boundary (the same assert as #1823) crashed the node mid-block-import. The crash left the node's ledger ParityDB in an internally inconsistent state: on every subsequent restart the node panics with root should be in the arena / key ... not in storage arena (the same failure as #1492) from both block import (post_block_update) and transaction validation (get_transaction_cost).

The node cannot recover on its own — it is stuck in a permanent crash loop (~900 panic log lines over 2h) and the only remediation is wiping the chain data / restoring the volume from a snapshot and resyncing.

Filing this because it links #1823 and #1492 causally: the GRANDPA qed-panic is not just a restart-and-carry-on crash; aborting mid-import can leave midnight-storage-core's arena missing its root key, which is fatal and persistent.

Environment

• Node version: 1.0.0-8af7d08a
• Network: Preprod
• Role: bootnode (stl-k8s-preprod-bootnode-02), Kubernetes StatefulSet, ParityDB backend
• Date: 2026-07-15, times below in UTC

Timeline

00:06–00:30 — The node's cardano-db-sync data source stalled (root cause on our side, confirmed after filing: the Cardano preprod epoch 300→301 rollover at 00:00 UTC made db-sync write an ~18-minute ledger snapshot that saturated its host's local disk; block ingestion into Postgres froze entirely 00:23–00:30). With the db tip stale, the node continuously failed mc_hash inherent verification for incoming blocks, banned peers, and stalled ~30 blocks behind the chain head at #1660714:

WARN partner_chains_db_sync_data_sources::block: Get stable block by hash failed: Block with hash 19421614... is not stable yet: block 4934966 requires latest block >= 4937126, but latest block is 4937124.
WARN midnight_node::inherent_data: Failed to create mc_hash inherent data for verification: Main chain state 19421614... referenced in imported block at slot 297345871 ... not found
WARN sync: 💔 Verification failed for block 0x0447ab48... received from (12D3KooW...): "Main chain state ... not found"

00:30:43 — db-sync recovered; the node burst-imported the ~30-block backlog (~10 blocks/s) with the finalizing justifications already in hand (status lines show finalized == best throughout the catch-up, vs. the normal 2-block gap on healthy nodes). The burst crossed the Midnight epoch boundary: ⏳ New epoch 991153 starting at block 1660744, block #1660744 (authority-set change) imported, then GRANDPA logged Re-finalized block #0xeedc... (1660744) in the canonical chain, current best finalized is #1660744.

00:30:45 — panic 1 (same assert as #1823):

Thread 'tokio-rt-worker' panicked at 'returns Ok when no authority set change should be enacted; qed;',
polkadot-sdk-.../2e4dd0b/substrate/client/consensus/grandpa/src/import.rs:859

Key frames:

6: sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC>::import_justification
7: <sc_consensus_grandpa::import::GrandpaBlockImport<...> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
10: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}

Process aborted, container restarted.

00:31:05 — panic 2, on first block import after restart, and on every restart since (same failure as #1492):

Thread 'tokio-rt-worker' panicked at 'root should be in the arena (T=midnight_node_ledger::ledger_8::common::api::ledger::Ledger<midnight_storage_core::db::paritydb::ParityDb>): Custom { kind: NotFound, error: "BackendLoader::get(): key 52109d3bb08548cffb0978527777c630b140c6575044e6196a2e0791436db0a7 not in storage arena. Are you sure you persisted this key or one of its ancestors?" }', midnight-storage-core-1.2.0/src/arena.rs:1662

Key frames (block-import path):

6: midnight_storage_core::arena::Sp<T,D>::force_as_arc
7: midnight_node_ledger::ledger_8::common::Bridge<S,D>::post_block_update
9: midnight_node_ledger::host_api::ledger_8::ledger_8_bridge::post_block_update_version_1
...
33: sp_api::Core::execute_block
34: <&sc_service::client::client::Client<...> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
36: <sc_consensus_grandpa::import::GrandpaBlockImport<...> as BlockImport<Block>>::import_block::{{closure}}

00:31 → ongoing — Crash loop. The same missing arena key 52109d3b... is hit from two paths, depending on what the node touches first after restart:

• block import: Bridge::post_block_update (frames above)
• transaction validation: Bridge::get_transaction_cost ← sc_transaction_pool ... validate_transaction_blocking

Each cycle the node starts, connects peers, imports a handful of blocks, then panics again — it never gets past the corrupted ledger state. ~900 panic log lines over the following two hours. Memory telemetry confirms it: the pod normally cycles 2–3.4 GiB; after 00:31 it flatlines at ~200 MB (early-startup footprint) with periodic restarts.

Downstream effect: our finalization-stall monitoring paged (P1) at 00:43.

Analysis (updated with fleet comparison — see first comment for detail)

1. The node: GRANDPA panic 'returns Ok when no authority set change should be enacted; qed' during justification import under network partitions #1823 GRANDPA assert is reachable in normal production operation, not just under Antithesis network-partition faults. Reproduction recipe: main-chain-follower (db-sync) stall → node falls N blocks behind → follower recovers → node burst-imports the backlog across an authority-set-change block while already holding the finalizing justification (finalized == best catch-up mode). The block import and justification import race; losing the race hits the assert.
2. It is a timing race, not deterministic. Two other nodes consuming the same stalled db-sync stalled at the same block (#1660714), burst-imported the same boundary block #1660744 in the same second, and survived — 1-in-3 hit rate in this cohort. A fourth node on a healthy follower imported #1660744 at the chain head, received the justification 12 s later, and applied the authority-set change through the normal path (👴 Applying authority set change scheduled at block #1660744), no warning.
3. Most importantly: aborting mid-import is not crash-safe for the ledger arena. The ledger root key referenced by the last (partially) imported block was never persisted to ParityDB, and after restart every ledger host call that resolves the root panics. There is no self-heal path — the node is bricked until its chain data is wiped or restored from a snapshot. This is what turns a lost race from a restart into an outage.
4. Suggested angles:
   • Make arena persistence atomic with (or ordered before) whatever state causes the node to consider the block imported, so a crash between the two can't strand a dangling root reference.
   • On startup, detect a missing/unresolvable ledger root and fail with an actionable error (or recover to the last consistent root) instead of panicking on first use.
   • Fix the node: GRANDPA panic 'returns Ok when no authority set change should be enacted; qed' during justification import under network partitions #1823 assert itself (upstream sc-consensus-grandpa); the justification-in-hand catch-up path above may narrow where to look.

Happy to provide full logs / the affected ParityDB volume contents to help debug.

[scottbuckel]

Update: fleet comparison — the panic is a timing race, and the trigger recipe is now precise

We compared every node that lived through the same window. Summary of what changed in the issue body (Timeline + Analysis revised accordingly):

1. The trigger chain is fully identified

The db-sync lag originally noted as a "contributing stressor" is the trigger. The Cardano preprod epoch 300→301 rollover (00:00 UTC) made cardano-db-sync write an ~18-minute ledger snapshot (Wrote snapshot ... in 1081.99s) that saturated its host's local disk (95–100% util, ~1,500–1,900 writes/s for 30 min); its block ingestion into Postgres froze entirely 00:23–00:30. (Postgres itself was healthy — flat ~0.2 ms commit latency — and cardano-node stayed at the Cardano chain tip throughout; the freeze was purely db-sync's.)

Downstream: with a stale db tip, midnight-node consumers failed mc_hash inherent verification, banned peers, and stalled ~30 blocks behind head. When db-sync recovered at 00:30:43, they burst-imported the backlog (~10 blocks/s) across the Midnight epoch-991153 authority-set-change block #1660744 — with the finalizing justifications already in hand (status lines show finalized == best during catch-up, vs. the normal 2-block finality gap).

2. Three nodes ran the identical race; one lost

node	same db-sync?	stalled at	imported #1660744	outcome
bootnode (this issue)	yes	#1660714	00:30:43.58	Re-finalized block warn → qed-panic 1.6 s later → arena corruption
validator A	yes	#1660714	00:30:43.68	survived, synced on
rpc node B	yes	#1660714	00:30:43.68	survived, synced on
bootnode on healthy db-sync	no	never stalled	00:30:00.37 (at head)	justification arrived 12 s later; 👴 Applying authority set change — clean

So: not deterministic — a race between block import and justification import during justification-in-hand catch-up, ~1-in-3 hit rate here. The crashed node is an archive bootnode (--state-pruning archive --blocks-pruning archive), which may widen the race window via heavier import work.

3. Why this matters for prioritization

• The node: GRANDPA panic 'returns Ok when no authority set change should be enacted; qed' during justification import under network partitions #1823 assert needs no fault injection: any main-chain-follower stall that ends with a catch-up burst across an authority-set change reproduces the setup. Cardano epoch rollovers make follower stalls a regular, schedulable event (every 5 days on preprod).
• The arena crash-unsafety (this issue) is the severity multiplier: for the node that lost the race, a would-be restart became permanent state corruption requiring a chain-data wipe.

A plausible reproduction without faults: run a node whose db-sync you pause (SIGSTOP or stop ingestion) for ~5–10 min across a Midnight epoch boundary, then resume — the node will catch up in finalized == best mode through the authority-set-change block. Repeat a handful of times to land the race.

[justinfrevert]

The write up makes sense. Might not make sense to change the Cardano observation parameters, but we should recover gracefully if these panics occur.

[scottbuckel]

Mechanism footnote for accuracy (does not change the node-side analysis): further investigation showed our db-sync host's disk saturation was memory thrash — the host (8 GB, shared by cardano-node + db-sync) had a fully-used 4 GiB swapfile on its root volume, and epoch-boundary processing tipped it into ~30 min of page-sized swap I/O that pegged the volume's IOPS ceiling; the 18-minute snapshot duration was a symptom of the thrash, not bulk writes.

For this issue nothing changes: the reproduction recipe stands as written — any mechanism that stalls the db-sync follower and then lets it recover produces the justification-in-hand catch-up burst across an authority-set change, and the SIGSTOP-the-follower repro suggested above remains the cleanest way to trigger it deliberately.

[ozgb]

Corruption can be avoided by running the node with STORAGE_SEPARATION=unified - see #1297