Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core, eth/downloader: implement pruning mode sync #31414

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

rjl493456442
Copy link
Member

@rjl493456442 rjl493456442 commented Mar 18, 2025

This pull request introduces new sync logic for pruning mode.

Pruning Mode:

Geth now supports two chain history modes (-history.chain): all and postmerge.

  • all (default): Retains the entire historical chain as usual.
  • postmerge: Discards chain segments before the configured cutoff point and skips them during synchronization.

The cutoff point is defined as the first block after the Paris fork and will be dynamically
adjusted in the future.


Sync mechanism with chain cutoff

A consensus has been reached across different EL implementations that chain
segments before the merge point will be pruned by default in the near future.
This means these segments—specifically block bodies and receipts—will no longer
be available in the P2P network. As a result, these components must be skipped
during retrieval in pruning mode.


Internally, Geth uses a freezer to maintain historical chain segments, which must
be inserted sequentially. This raises the question of how to handle the gap between
the genesis block and the first block after the cutoff in the freezer.

Two options were considered:
(a) Introducing a new freezer API to set the freezer tail to an arbitrary block number.
(b) Replacing chain data before the cutoff with null data.

The latter option is simpler and functionally equivalent to the former. When retrieving
data for chain segments before the cutoff, null values will be returned, indicating that
these items do not exist locally. Additionally it's a more robust approach that all chain
segment inserted into the freezer always in an atomic manner. Lots of complexities are
needed in approach (a) if this property is required. For example, we must guarantee
the header x write and body table tail shifting to x happen atomically.

Therefore, the approach (b) is chosen for the sake of simplicity.


Additionally, this pull request introduces some important changes worth highlighting.

Originally, in snap sync, the header chain and other components (bodies and receipts)
were inserted separately. However, in Proof-of-Stake, this separation is unnecessary
since the sync target is already verified by the CL.

To simplify the process, this pull request modifies InsertReceiptChain to insert headers
along with block bodies and receipts together. Besides, InsertReceiptChain doesn't
have the notion of reorg, as the common ancestor is always be found before the sync
and extra side chain is truncated at the beginning if they fall in the ancient store.
The stale canonical chain flags will always be rewritten by the new chain. No explicit
reorg logic required in the InsertReceiptChain.

@rjl493456442 rjl493456442 requested a review from holiman as a code owner March 18, 2025 07:19
@rjl493456442 rjl493456442 force-pushed the prunemode-sync-2 branch 3 times, most recently from 916eaf0 to d3b8b71 Compare March 20, 2025 07:07
@rjl493456442 rjl493456442 changed the title [WIP] core, eth/downloader: implement pruning mode sync core, eth/downloader: implement pruning mode sync Mar 20, 2025
@fjl fjl added this to the 1.15.6 milestone Mar 20, 2025
@rjl493456442 rjl493456442 force-pushed the prunemode-sync-2 branch 2 times, most recently from 386d78f to 5dd27fd Compare March 20, 2025 12:57
@fjl fjl removed this from the 1.15.6 milestone Mar 20, 2025
@rjl493456442
Copy link
Member Author

Snap sync with --chain.history=postmerge

+-----------------------+---------------------------+------------+------------+
|       DATABASE        |         CATEGORY          |    SIZE    |   ITEMS    |
+-----------------------+---------------------------+------------+------------+
| Key-Value store       | Headers                   | 51.39 KiB  |         81 |
| Key-Value store       | Bodies                    | 6.05 MiB   |         81 |
| Key-Value store       | Receipt lists             | 6.33 MiB   |         81 |
| Key-Value store       | Difficulties (deprecated) | 0.00 B     |          0 |
| Key-Value store       | Block number->hash        | 3.32 KiB   |         81 |
| Key-Value store       | Block hash->number        | 863.83 MiB |   22092473 |
| Key-Value store       | Transaction index         | 13.39 GiB  |  388581603 |
| Key-Value store       | Bloombit index            | 4.57 GiB   |   11050258 |
| Key-Value store       | Log search index          | 80.00 B    |          3 |
| Key-Value store       | Contract codes            | 9.62 GiB   |    1545294 |
| Key-Value store       | Hash trie nodes           | 0.00 B     |          0 |
| Key-Value store       | Path trie state lookups   | 96.17 KiB  |       2402 |
| Key-Value store       | Path trie account nodes   | 44.69 GiB  |  387079001 |
| Key-Value store       | Path trie storage nodes   | 172.86 GiB | 1720525876 |
| Key-Value store       | Verkle trie nodes         | 0.00 B     |          0 |
| Key-Value store       | Verkle trie state lookups | 0.00 B     |          0 |
| Key-Value store       | Trie preimages            | 0.00 B     |          0 |
| Key-Value store       | Account snapshot          | 12.96 GiB  |  283406617 |
| Key-Value store       | Storage snapshot          | 91.63 GiB  | 1269738120 |
| Key-Value store       | Beacon sync headers       | 1.21 KiB   |          2 |
| Key-Value store       | Clique snapshots          | 0.00 B     |          0 |
| Key-Value store       | Singleton metadata        | 695.72 KiB |         16 |
| Light client          | CHT trie nodes            | 0.00 B     |          0 |
| Light client          | Bloom trie nodes          | 0.00 B     |          0 |
| Ancient store (Chain) | Headers                   | 10.54 GiB  |    6555000 |
| Ancient store (Chain) | Hashes                    | 800.62 MiB |    6555000 |
| Ancient store (Chain) | Bodies                    | 385.42 GiB |    6555000 |
| Ancient store (Chain) | Receipts                  | 120.87 GiB |    6555000 |
+-----------------------+---------------------------+------------+------------+
|                                   TOTAL           | 868.21 GIB |            |
+-----------------------+---------------------------+------------+------------+

@rjl493456442
Copy link
Member Author

Snap sync with --chain.history=all

+-----------------------+---------------------------+------------+------------+
|       DATABASE        |         CATEGORY          |    SIZE    |   ITEMS    |
+-----------------------+---------------------------+------------+------------+
| Key-Value store       | Headers                   | 55.82 KiB  |         88 |
| Key-Value store       | Bodies                    | 6.42 MiB   |         88 |
| Key-Value store       | Receipt lists             | 6.73 MiB   |         88 |
| Key-Value store       | Difficulties (deprecated) | 0.00 B     |          0 |
| Key-Value store       | Block number->hash        | 3.61 KiB   |         88 |
| Key-Value store       | Block hash->number        | 863.83 MiB |   22092480 |
| Key-Value store       | Transaction index         | 13.39 GiB  |  388578627 |
| Key-Value store       | Bloombit index            | 4.57 GiB   |   11050258 |
| Key-Value store       | Log search index          | 203.38 MiB |    2135822 |
| Key-Value store       | Contract codes            | 9.62 GiB   |    1545294 |
| Key-Value store       | Hash trie nodes           | 0.00 B     |          0 |
| Key-Value store       | Path trie state lookups   | 76.79 KiB  |       1918 |
| Key-Value store       | Path trie account nodes   | 44.69 GiB  |  387071252 |
| Key-Value store       | Path trie storage nodes   | 172.86 GiB | 1720536779 |
| Key-Value store       | Verkle trie nodes         | 0.00 B     |          0 |
| Key-Value store       | Verkle trie state lookups | 0.00 B     |          0 |
| Key-Value store       | Trie preimages            | 0.00 B     |          0 |
| Key-Value store       | Account snapshot          | 12.96 GiB  |  283408241 |
| Key-Value store       | Storage snapshot          | 91.63 GiB  | 1269744950 |
| Key-Value store       | Beacon sync headers       | 619.00 B   |          1 |
| Key-Value store       | Clique snapshots          | 0.00 B     |          0 |
| Key-Value store       | Singleton metadata        | 394.43 MiB |         19 |
| Light client          | CHT trie nodes            | 0.00 B     |          0 |
| Light client          | Bloom trie nodes          | 0.00 B     |          0 |
| Ancient store (Chain) | Receipts                  | 237.47 GiB |   22092393 |
| Ancient store (Chain) | Headers                   | 10.54 GiB  |   22092393 |
| Ancient store (Chain) | Hashes                    | 800.62 MiB |   22092393 |
| Ancient store (Chain) | Bodies                    | 629.05 GiB |   22092393 |
| Ancient store (State) | History.Meta              | 147.75 KiB |       1915 |
| Ancient store (State) | Account.Index             | 17.96 MiB  |       1915 |
| Ancient store (State) | Storage.Index             | 22.97 MiB  |       1915 |
| Ancient store (State) | Account.Data              | 19.03 MiB  |       1915 |
| Ancient store (State) | Storage.Data              | 7.55 MiB   |       1915 |
+-----------------------+---------------------------+------------+------------+
|                                   TOTAL           |  1.20 TIB  |            |
+-----------------------+---------------------------+------------+------------+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants