fix(s3): R2 Copy compatibility + Remove sync checks #38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
samcm
commented
Mar 18, 2025
•
samcm
commented
- Removes sync checks based on the node status. This is one of the learnings from Holesky. We can't trust the beacon node to accurately report it's sync status, and would prefer to over-index data instead of dropping it. We now take a different approach, comparing the wall clock slot with the slot from the event stream to determine if we should fetch data.
- Fixes permanent block storage on R2, which requires a download/upload instead of a server-side copy
This commit introduces several enhancements to the block storage mechanism and improves compatibility with S3-like storage solutions, particularly R2. - Adds the 'Slot' field to the 'PermanentStoreBlock' struct and uses it for organizing blocks in the permanent storage location. - Modifies the S3 store to properly URL-escape the source path during copy operations, ensuring compatibility with R2 and other S3-like services. - Updates the indexer to include the block slot when queuing blocks for permanent storage and purging old blocks.
This change introduces a sync tolerance mechanism to the agent, allowing it to ignore events from slots that are too far behind the current head. This prevents the agent from fetching blocks that are unlikely to be relevant. The sync tolerance is configurable via the `syncToleranceSlots` parameter in the Ethereum configuration. The `ShouldIgnoreEventFromSlot` method in the `Node` struct checks if an event's slot is within the tolerance window. fix(store): fix S3 copy source encoding for R2 compatibility The S3 copy source encoding was not properly handling URL escaping, leading to issues with R2 compatibility. This change ensures that the copy source is correctly encoded, allowing for seamless operation with R2 and other S3-compatible services.
This commit introduces a fallback mechanism for the S3Store's Copy function to support R2 storage, which has limitations with server-side copy operations. When a regular server-side copy fails, the code now attempts to perform a manual copy by first retrieving the source object using GetObject, then uploading it to the destination using PutObject. This workaround ensures compatibility with R2 and other S3-compatible services that may not fully support server-side copy. The code also preserves metadata during the GetObject+PutObject fallback.
samcm
changed the title
This commit introduces a metrics server for the agent and a create beacon block endpoint. The agent now exposes metrics via a metrics server, which can be configured via the MetricsAddr config option. The create beacon block endpoint allows clients to create new beacon blocks.
This commit introduces a new database model, `PermanentBlock`, to track blocks that have been copied to permanent storage. It also modifies the permanent store logic to record blocks in the database after they are copied. The permanent location is changed to not include the slot in the path.
The linter was complaining about the use of int64 for slot. Added //nolint:gosec to the lines that were causing the issue.
This commit refactors the block processing logic to check if a block already exists in the database before proceeding with further processing. This change optimizes the workflow by avoiding unnecessary operations on already recorded blocks.
refactor(persistence): remove redundant initialization of values slice in DistinctPermanentBlockValues
…when getting a non-existent block
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.