tracing regression in erigon 3

[banteg]

System information

Erigon version: erigon version 3.00.0-dev-be889f62

OS & Version: Linux

Commit hash: be889f6

Erigon Command (with flags/config):

Consensus Layer: erigon internal

Consensus Layer Command (with flags/config):

Chain/Network: ethereum

Expected behaviour

correct traces are returned

Actual behaviour

rpc errors out with "insufficient funds for gas", as well as "nonce too low".

this also affects receipts since erigon now generates them ad hoc.

Steps to reproduce the behaviour

i ran cryo traces -a on a full block range, it has reported errors in 23 chunks (23k blocks).

then i further isolated the issue by running trace_block against these block ranges, and narrowed it down to 6105 blocks.

then i ran trace_transaction against every transaction in these blocks and wrote down the ones that have returned an error. however, i noticed that all the txs after the first failing transaction are failing. so i'm only providing the first failing tx for each block i got from trace_block.

the full list can be found here. it should be helpful for producing and verifying the fix.

[awskii]

thank you for valid test, trying to reproduce now, could take day or two.
Previously such errors happened during HisotryExecution and most of the time meant that bad history file provided. But i assume this is happening on available snapshots on webseeds, so i am syncing them now.

UPD: issue persists and roots down to incorrect snapshot .ef files for given region. Log was helpful - some blocks are going by long increasing ranges which gave a clue.

[AskAlexSharov]

fixed by #10746 but still we need need release 2 new .ef files.
also removing historical state cache reduced perf, created ticket to re-introduce cache #10747

[AskAlexSharov]

fixed. but need (git pull and re-sync). and we didn't work on #10747 yet

[banteg]

this appears fixed as of 900f730