chain out of sync

[fatminer]

I didn't update my nodes when the recent hard fork took place.
Because I was following another forked chain, I started from the snapshot, but the synchronization of the differences does not finish forever.

2025-02-15T07:53:15.847Z
INFO [02-15|07:53:15.847] Imported new chain segment number=67,943,548 hash=7e6cf2..85f1f6 blocks=1 txs=75 mgas=11.977 elapsed=14.278s mgasps=0.839 age=1h9m49s snapdiffs=6.27MiB triedirty=78.66MiB
2025-02-15T07:53:17.054Z
INFO [02-15|07:53:17.054] Whitelisting checkpoint deferred err="chain out of sync"
2025-02-15T07:53:24.606Z
INFO [02-15|07:53:24.606] Whitelisting milestone deferred err="chain out of sync"

Is this because the Polygon network is broken?
It seems that many nodes are out of sync.

I can't find an answer even if I ask on Discord.
It would be helpful if you could show me a clear solution.

[valamidev]

Same issue here. Sync via the open network is extremely slow.

[valamidev]

With Bor v2.0.0 sync got significant faster.

[fatminer]

I updated to bor v2.0.0, but the situation did not change.
It is recommended to change the DB engine, but this is not possible as it is necessary to utilize the leveldb data that has been synchronized so far.

Feb 18 06:43:04 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:04.912] Whitelisting milestone deferred          err="chain out of sync"
Feb 18 06:43:16 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:16.913] Whitelisting milestone deferred          err="chain out of sync"
Feb 18 06:43:16 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:16.913] Whitelisting checkpoint deferred         err="chain out of sync"

Also, since official snapshots have not been released, you will need to use the ones provided by the servers and parties.
I don't think it's a good idea to trust a third party.

[valamidev]

I updated to bor v2.0.0, but the situation did not change. It is recommended to change the DB engine, but this is not possible as it is necessary to utilize the leveldb data that has been synchronized so far.

Feb 18 06:43:04 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:04.912] Whitelisting milestone deferred          err="chain out of sync"
Feb 18 06:43:16 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:16.913] Whitelisting milestone deferred          err="chain out of sync"
Feb 18 06:43:16 ip-10-105-0-4 docker-compose[55029]: #033[36mbor             |#033[0m INFO [02-18|06:43:16.913] Whitelisting checkpoint deferred         err="chain out of sync"

Also, since official snapshots have not been released, you will need to use the ones provided by the servers and parties. I don't think it's a good idea to trust a third party.

There is no known risk to using 3rd party snapshots, one of the core feature of a blockchain node is to verify the integrity and state with the chain(other peers).

[wdsse]

I have the same problem, is there a solution for this problem?
After restarting, synchronization works, but after a while it stops synchronizing again

[fatminer]

As far as I'm experimenting.

・Uses bor 2.0.0 / heimdall 1.2.0
・Make db.engine pebble
・Extract the pebble snapshot

With the above steps, synchronization is achieved.
However, I think the problem is that the official company does not provide snapshots.
And this issue has been a hot topic in the Discord community.

Don't Trust. Verify.

[shora0]

Why is the snapshot I downloaded from https://publicnode.com/snapshots#polygon unable to recognize the snapshot’s blocks and needs to be resynchronized?

config.toml is set:
“db.engine” = “pebble”
"state.scheme" = "path"

It does not seem to pick up this setting in the config toml.

[marcello33]

Hey @shora0 have you verified with other snapshots? Maybe that's corrupted?
As per @fatminer comment, with those steps it should work.
We are no longer providing "official" snapshots

[shora0]

Only the base snapshot of https://publicnode.com/snapshots#polygon can be downloaded and then resynced to work properly.

[shora0]

But it also needs to be resynchronized to generate the Pebble data. Pebble snapshots for sections using https://publicnode.com/snapshots#polygon are not available. When importing, the error "The ancient link segment has been extracted, please set --datadir.ancient to the correct path" appears, and bor cannot run normally.

[johnnyji-dev]

I have a same issue.
Already update latest version (bor 2.0.0 & hemidal 1.2.0).

After syncing bor block, it stopped syncing block.
The log message of "./bor server .." is like below

...
INFO [02-21|02:44:09.394] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:44:21.395] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:44:33.394] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:44:45.394] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:44:57.394] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:45:09.395] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:45:21.395] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-21|02:45:33.394] Whitelisting milestone deferred          err="chain out of sync"
...

BTW, if kill the 'bor' process and then restart the process, it can sync the block.
But after syncing, it stop syncing again.

[marcello33]

@manav2401 is this something you're aware of?

[wdsse]

The problem is still relevant. here are the logs:

INFO [02-25|10:43:02.286] Fetching state updates from Heimdall     fromID=3,029,411 to=2025-02-25T10:40:33Z
INFO [02-25|10:43:02.286] Fetching state sync events               queryParams="from-id=3029411&to-time=1740480033&limit=50"
INFO [02-25|10:43:02.287] Fetching state updates from Heimdall     fromID=3,029,411 to=2025-02-25T10:40:33Z
INFO [02-25|10:43:02.287] Fetching state sync events               queryParams="from-id=3029411&to-time=1740480033&limit=50"
INFO [02-25|10:43:02.288] StateSyncData                            gas=0         number=68,351,792 lastStateID=3,029,410 "total records"=0  "fetch time"=2  "process time"=0
INFO [02-25|10:43:02.288] StateSyncData                            gas=0         number=68,351,792 lastStateID=3,029,410 "total records"=0  "fetch time"=2  "process time"=0
INFO [02-25|10:43:03.625] Imported new chain segment               number=68,351,797 hash=731385..2512ff blocks=6    txs=514     mgas=81.382  elapsed=1.485s    mgasps=54.778 snapdiffs=4.60MiB triedirty=255.31MiB
INFO [02-25|10:43:25.945] Whitelisting new milestone from heimdall block=68,351,785 hash=c7d4c9..4f9f3e
INFO [02-25|10:44:25.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:44:37.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:44:49.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:45:01.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:45:13.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:45:25.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:45:37.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:45:49.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:46:01.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:46:13.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:46:25.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:46:37.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:46:49.942] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:47:01.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:47:13.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:47:25.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:47:37.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:47:49.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:48:01.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:48:13.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:48:25.946] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:48:37.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:48:49.946] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:49:01.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:49:13.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:49:25.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:49:37.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:49:49.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:50:01.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:50:13.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:50:25.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:50:37.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:50:49.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:01.942] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:13.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:25.943] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:37.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:49.944] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:51:53.959] Whitelisting new checkpoint from heimdall block=68,350,885 hash=b63735..d21c01
INFO [02-25|10:52:01.946] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:52:13.946] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:52:25.958] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:52:37.945] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-25|10:52:49.947] Whitelisting milestone deferred          err="chain out of sync"

Without any errors the software stops synchronizing

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[manav2401]

Hi, can you share debug logs (verbosity set to 4) if possible as the info available above isn't enough to debug. Logs when bor was syncing and lost sync should be helpful. Also, as Leon commented, please make sure you're providing datadir and ancient datadir paths correctly and are extracting the snapshots correctly? You can use path + pebble db snapshot for faster sync performance.