op-supervisor,op-node: Exhaust L1 stalled loop

[sebastianst]

During the interop rehearsal we came across a strange stalled loop between one op-supervisor and op-node (that was driving a reth node) where it just didn't do any derivation progress and was stuck at the same L1 block. We only have info logs:

op-supervisor:

t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120009-1 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-0-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120009 l2=0x3ea79e419b5337c92873c21b867e09f5f50cac541c9f63b65c87c2770339e811:39170 l1=0x6ba8958660cb1fb727db53f17d85b73874337bb9723ee482b6154d0617db72d9:8477038
t=2025-06-04T19:01:19+0000 lvl=debug msg="Next L1 block is not yet available" syncnode=syncnode-420120009-1 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-0-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120009 l1Block=0x6ba8958660cb1fb727db53f17d85b73874337bb9723ee482b6154d0617db72d9:8477038 err="not found"
t=2025-06-04T19:01:19+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xba0963e26a505b363882a50aaa2692f69ff38c6526b8425d05b0aa3e6f0d2148:8473340

I also included one Node completed syncing request by the other node it was managing, which was making progress just fine.

op-node:

t=2025-06-04T19:00:16+0000 lvl=warn msg="Received signal for L1 block, but needed different block" current=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 next=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040
t=2025-06-04T19:00:16+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=warn msg="Received signal for L1 block, but needed different block" current=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 next=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040
t=2025-06-04T19:00:16+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=warn msg="Received signal for L1 block, but needed different block" current=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 next=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040
t=2025-06-04T19:00:16+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341
t=2025-06-04T19:00:16+0000 lvl=warn msg="Received signal for L1 block, but needed different block" current=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341 next=0x449328840338733e94a21b0980513cfd061e1e1a52ee673f3fe864882f27c0ed:8473341

In a loop stuck at the same block.

A restart of the op-node fixed the problem initially, it did a reset and then synced fine again. However, after a few minutes it started becoming very slow again.

[sebastianst]

We're adding some logs with #16285 to get better insights into this.

[sebastianst]

The issue is caused by duplicate Exhaust L1 events that are triggered by StepReqEvents, which in turn are triggered inside SyncDeriver.onIncomingP2PBlock when unsafe blocks are queued. Many exhaust l1 events would arrive at the supervisor, which would respond many times with the same answer, causing the "Received signal for L1 block, but needed different block" error message. Now each of those would also trigger another step, so an avalanche of ever more duplicate events is sent. Now very slowly progress is made because at some point a new L1 block is sent back to the op-node. But by that time, even more exhaust L1 requests have been sent, making progress slower and slower, caused by the avalanche of exhaust L1 requests.

Reference logs of when it all started (internal)

• rehearsal-0 snapsync-2 op-supervisor
• rehearsal-0-bn-1 snapsync-2 op-node

op-supervisor:

2025-06-04 18:29:14.571 | t=2025-06-04T18:29:14+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x19cb758efc3882e3abe25d15fccf7ec7dd5bac99e5b5da5edad2ae2f6938e193:8473325 |  
-- | -- | --
  |   | 2025-06-04 18:29:14.758 | t=2025-06-04T18:29:14+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 |  
  |   | 2025-06-04 18:29:14.951 | t=2025-06-04T18:29:14+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 |  
  |   | 2025-06-04 18:29:15.061 | t=2025-06-04T18:29:15+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 |  
  |   | 2025-06-04 18:29:15.372 | t=2025-06-04T18:29:15+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 |  
  |   | 2025-06-04 18:29:16.091 | t=2025-06-04T18:29:16+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 |  
  |   | 2025-06-04 18:29:16.504 | t=2025-06-04T18:29:16+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 |  
  |   | 2025-06-04 18:29:16.763 | t=2025-06-04T18:29:16+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:16.778 | t=2025-06-04T18:29:16+0000 lvl=info msg="Updated latest known L1 block" service=l1-processor ref=0x38fb0bf0106ba134dddcc45a2419558cf2040901d0c8e96b294dc50c2568c968:8476880 |  
  |   | 2025-06-04 18:29:16.848 | t=2025-06-04T18:29:16+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:16.927 | t=2025-06-04T18:29:16+0000 lvl=debug msg="Updated finalized L1" finalizedL1=0x1744e4fe7444f9991da32aafbcc730a37eaf52c6ba71e2743cf62b70ce1ba1f3:8476814 |  
  |   | 2025-06-04 18:29:16.927 | t=2025-06-04T18:29:16+0000 lvl=debug msg="Updated finalized L1" finalizedL1=0x1744e4fe7444f9991da32aafbcc730a37eaf52c6ba71e2743cf62b70ce1ba1f3:8476814 |  
  |   | 2025-06-04 18:29:16.948 | t=2025-06-04T18:29:16+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.071 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.166 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.177 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.366 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.377 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.547 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.562 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:17.570 | t=2025-06-04T18:29:17+0000 lvl=info msg="Node completed syncing" syncnode=syncnode-420120010-2 endpoint=RPCSyncSource(ws://op-node.rehearsal-0-bn-1-opn-reth-f-snapsync-2.svc.cluster.local:9645) chain=420120010 l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331

op-node:

2025-06-04 18:29:14.550 | t=2025-06-04T18:29:14+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65 number=34040 state_root=0xc8d34452551ffbfa8ca2da00d1a4d52627802f41dd457be3332f2d119de224eb timestamp=1749019004 parent=0x20ca5c57f7468a97abfc6af82c753bf25c19190411dfc1a6e270984ab9baede9 prev_randao=0x45308083711a353915f5c718213fb28354f1ef3c13c665eb8f92134073c41aa8 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=20.218ms insert_time=35.817ms total_time=56.035ms mgas=0.051696 mgasps=0.9225517815112707 |  
-- | -- | --
  |   | 2025-06-04 18:29:14.550 | t=2025-06-04T18:29:14+0000 lvl=info msg="Record safe head" l2=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 l1=0x19cb758efc3882e3abe25d15fccf7ec7dd5bac99e5b5da5edad2ae2f6938e193:8473325 |  
  |   | 2025-06-04 18:29:14.570 | t=2025-06-04T18:29:14+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x19cb758efc3882e3abe25d15fccf7ec7dd5bac99e5b5da5edad2ae2f6938e193:8473325 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:14.668 | t=2025-06-04T18:29:14+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 |  
  |   | 2025-06-04 18:29:14.757 | t=2025-06-04T18:29:14+0000 lvl=info msg="Derivation continued with next L1 block" current=0x19cb758efc3882e3abe25d15fccf7ec7dd5bac99e5b5da5edad2ae2f6938e193:8473325 next=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 |  
  |   | 2025-06-04 18:29:14.757 | t=2025-06-04T18:29:14+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:14.758 | t=2025-06-04T18:29:14+0000 lvl=info msg="Advancing bq origin" origin=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 originBehind=false |  
  |   | 2025-06-04 18:29:14.758 | t=2025-06-04T18:29:14+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:14.766 | t=2025-06-04T18:29:14+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 |  
  |   | 2025-06-04 18:29:14.950 | t=2025-06-04T18:29:14+0000 lvl=info msg="Derivation continued with next L1 block" current=0xeed2f5d3a93e521bfa3e7b17cd6cc3dd862da47fb3af7575a9a77ffa7169120c:8473326 next=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 |  
  |   | 2025-06-04 18:29:14.950 | t=2025-06-04T18:29:14+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:14.950 | t=2025-06-04T18:29:14+0000 lvl=info msg="Advancing bq origin" origin=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 originBehind=false |  
  |   | 2025-06-04 18:29:14.950 | t=2025-06-04T18:29:14+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:14.959 | t=2025-06-04T18:29:14+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x83cec6d55648b27a66be8b16dac90758665d1f142cd18408e7dcff3a6f3daa3c:76791 |  
  |   | 2025-06-04 18:29:14.959 | t=2025-06-04T18:29:14+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:15.061 | t=2025-06-04T18:29:15+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 |  
  |   | 2025-06-04 18:29:15.264 | t=2025-06-04T18:29:15+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 |  
  |   | 2025-06-04 18:29:15.371 | t=2025-06-04T18:29:15+0000 lvl=info msg="Derivation continued with next L1 block" current=0xcdd24531df19696eef02b18813729213623dddfdae60a4e8146daea41d93657f:8473327 next=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 |  
  |   | 2025-06-04 18:29:15.371 | t=2025-06-04T18:29:15+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:15.371 | t=2025-06-04T18:29:15+0000 lvl=info msg="Advancing bq origin" origin=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 originBehind=false |  
  |   | 2025-06-04 18:29:15.371 | t=2025-06-04T18:29:15+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:15.561 | t=2025-06-04T18:29:15+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 |  
  |   | 2025-06-04 18:29:16.090 | t=2025-06-04T18:29:16+0000 lvl=info msg="Derivation continued with next L1 block" current=0xb107ee4b55a9398725c4988deb4516f44ab6567007737fbc5c77d9054fe25443:8473328 next=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 |  
  |   | 2025-06-04 18:29:16.090 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xfb56ab7b120ccf53cde5ce7280ba6199c6cf1bd52e837049f5a644ded147d60e:76792 |  
  |   | 2025-06-04 18:29:16.090 | t=2025-06-04T18:29:16+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:16.090 | t=2025-06-04T18:29:16+0000 lvl=info msg="Advancing bq origin" origin=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 originBehind=false |  
  |   | 2025-06-04 18:29:16.090 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.152 | t=2025-06-04T18:29:16+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 |  
  |   | 2025-06-04 18:29:16.504 | t=2025-06-04T18:29:16+0000 lvl=info msg="Derivation continued with next L1 block" current=0x445a24ae29aba0b5c126441c3ab3634084b14f63ca5275c750d6f2314a79d9ae:8473329 next=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 |  
  |   | 2025-06-04 18:29:16.504 | t=2025-06-04T18:29:16+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:16.504 | t=2025-06-04T18:29:16+0000 lvl=info msg="Advancing bq origin" origin=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 originBehind=false |  
  |   | 2025-06-04 18:29:16.504 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.553 | t=2025-06-04T18:29:16+0000 lvl=info msg="Received next L1 block" mode=managed chainId=420120010 nextL1=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:16.762 | t=2025-06-04T18:29:16+0000 lvl=info msg="Derivation continued with next L1 block" current=0x80627abeafcf68b27aa48b9956579a42744c1b2219b339e8dd2e03ee3abb75c9:8473330 next=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 |  
  |   | 2025-06-04 18:29:16.762 | t=2025-06-04T18:29:16+0000 lvl=info msg="Transforming stages" fork=none |  
  |   | 2025-06-04 18:29:16.762 | t=2025-06-04T18:29:16+0000 lvl=info msg="Advancing bq origin" origin=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 originBehind=false |  
  |   | 2025-06-04 18:29:16.762 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.762 | t=2025-06-04T18:29:16+0000 lvl=info msg="processing L2 range request" target=34040 end=0xf441dd2d81e9dc62a952259f3bbcdea0cacfc9afcd3cd6bc5b4b57ee29fdba86:72765 rangeReqId=1 |  
  |   | 2025-06-04 18:29:16.763 | t=2025-06-04T18:29:16+0000 lvl=info msg="no peers ready to handle block requests for more P2P requests for L2 block history" target=34040 end=0xf441dd2d81e9dc62a952259f3bbcdea0cacfc9afcd3cd6bc5b4b57ee29fdba86:72765 current=72628 |  
  |   | 2025-06-04 18:29:16.765 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x2d742992e5d3d1f82b555b410beb53fc3c193a593a9a193339c8a54c0d35bba1:72764 |  
  |   | 2025-06-04 18:29:16.765 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xf6bf30f332ecc3d8841a418df6474c584a0eea6fa133db1b352195ae009b0840:72763 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xa943e9220b9076bc70cf0ce55cfd63e508063c4842e48837b5ffaad290167a78:72762 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x669ae11661d9f7d6f5f3168a6f9dbf3676af7aa5c5156605247e776c59f478b7:72761 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x07f9884a58688fa4c8259c1c3a2145b1c784cac63db10d6c4b43f881d3d7b0e5:72760 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x7222a3516e3f6e02e24c5490d5d108d77bd71b52a158752a87f0f699203e650c:72759 |  
  |   | 2025-06-04 18:29:16.766 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.767 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x40c4a79cd04d6c21cf25bffe7984c4116283a3e0c5b1f1a9535c3f2c59322c3a:72758 |  
  |   | 2025-06-04 18:29:16.767 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x639ba55e545a45222ea9f82eb25b26366f69ff82ef109d147cf617dc3d161a90:72757 |  
  |   | 2025-06-04 18:29:16.768 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.769 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x8a733ccab854b7b226c43e9174b457f625c6a8c8baf71bc564aa40ffaad88c02:72756 |  
  |   | 2025-06-04 18:29:16.770 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.771 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x84ad34e26e81a438b49c897e71611b7d80d7a6c7b586cca2ff1305e1b2e55e1b:72755 |  
  |   | 2025-06-04 18:29:16.772 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.772 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x01973998fc28259f575734d2f07a4a448692c8c83732501251e391e59434171b:72754 |  
  |   | 2025-06-04 18:29:16.773 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.774 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xb2b79689ebacac39e7aa9aa03d4e09877cb68d89c9621cf58cc09e94ab260b18:72753 |  
  |   | 2025-06-04 18:29:16.774 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.776 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x34f8388f638245949773198ace1d69c057e491ef78d35490870c4a8f7cfb8022:72752 |  
  |   | 2025-06-04 18:29:16.776 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.777 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xff9db88d60936d8d1937dca390042b16e02acf4f03a9d31d8a0d4240634bd6b8:72751 |  
  |   | 2025-06-04 18:29:16.778 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.779 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x0c4b549ab5d783b6f305bc2a16a1e2b5ab71ef1b0f6cbb1ae077be64d81ef646:72750 |  
  |   | 2025-06-04 18:29:16.779 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.780 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xdb36e61746a500a6914fd8b1ad1026c9f7f7d2d6bf85379907183ae5c0aeb9c7:72749 |  
  |   | 2025-06-04 18:29:16.780 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.782 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x134de9c3c7b4b6b528ed3bf663162e55eee0481f1f8723c56917aa1929c7e8f9:72748 |  
  |   | 2025-06-04 18:29:16.783 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x72bd41899edb29e6d49e92188ab49a18ae77dbc7adb722973cff1ad4a158f1f9:72747 |  
  |   | 2025-06-04 18:29:16.783 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.783 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x933a258fe9733d96beccfe62518ca89f71be2b37454fe11e8d1b4f6861e68e98:72746 |  
  |   | 2025-06-04 18:29:16.784 | t=2025-06-04T18:29:16+0000 lvl=info msg="Exhausted L1 data" mode=managed chainId=420120010 derivedFrom=0x687dc7eb4b8a5aaf02f48179338fadab2938f2c402c3ac768b84f1f7ee592126:8473331 derived=0x1ae011189eb33430aecdfb56537d8926da93d18f87437a6213f4154ac349bb65:34040 |  
  |   | 2025-06-04 18:29:16.785 | t=2025-06-04T18:29:16+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xb14d35388c95f96697ba46e087198539da299cd0e312cd8ac251e2c6549dadcf:72745