Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup node manager #21

Merged
merged 52 commits into from
Feb 26, 2025
Merged

Rollup node manager #21

merged 52 commits into from
Feb 26, 2025

Conversation

frisitano
Copy link
Collaborator

No description provided.

Copy link
Collaborator Author

@frisitano frisitano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments inline.

@@ -117,7 +117,7 @@ jobs:
include:
- type: wasm
target: wasm32-unknown-unknown
exclude: engine,scroll-wire,scroll-bridge,scroll-network
exclude: scroll-engine,scroll-wire,scroll-bridge,scroll-network,scroll-rollup-node
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently all our creates are being skipped.

alloy-rpc-types-engine = { version = "0.11.0", default-features = false }

# scroll-alloy
scroll-alloy-provider = { git = "https://github.com/scroll-tech/reth.git", branch = "feat/add-deref-blanket-engine-api" }
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before merge we should merge feat/add-deref-blanket-engine-api into scroll and then revert back to default on these dependency definitions.

@@ -0,0 +1,228 @@
{
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently blocks 2 - 5 can not be read as this dump does not encode the data using serde. It is instead sourced from the rpc response using cast. We should replace this with serde data such that we can deserialize in testing.


# scroll
reth-scroll-primitives = { workspace = true, features = ["serde"] }

# misc
eyre.workspace = true
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?

@frisitano frisitano marked this pull request as ready for review February 12, 2025 17:58
@frisitano frisitano requested review from Thegaram and greged93 and removed request for Thegaram February 12, 2025 17:58
Copy link
Collaborator

@greged93 greged93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I think the refactor of the Networking make the code flow clearer! Just couple of nits and questions

@@ -110,7 +102,7 @@ mod tests {
}

#[test]
fn test_matching_payloads() -> Result<()> {
fn test_matching_payloads() -> eyre::Result<()> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of custom errors you mean? I don't have a strong opinion on this, if it simplifies the code let's use eyre everywhere

Copy link
Collaborator

@greged93 greged93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a remark on the engine, otherwise lgtm!

let payload_status = self.new_payload(execution_payload).await?;

// Invoke the FCU with the new state.
let fcu = self.forkchoice_updated(fcs, None).await?;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we do fcs.safe_block_hash = unsafe_block_info.hash; before issuing the forkchoice?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic below defines what is currently implemented to manage fcs. We get the current fcs from the node manager and then set the head block hash to the block hash of the unsafe block received from the network when defining the future for importing the block. Do you think we should be setting the safe block hash when we receive a new block over p2p?

// Send the block to the engine to validate the correctness of the block.
let mut fcs = self.get_alloy_fcs();
let engine = self.engine.clone();
let future = Box::pin(async move {
trace!(target: "scroll::node::manager", "handling block import future for block {:?}", block.hash_slow());
// convert the block to an execution payload and update the forkchoice state
let execution_payload: ExecutionPayload =
ExecutionPayloadV1::from_block_slow(&block).into();
let unsafe_block_info: BlockInfo = (&execution_payload).into();
fcs.head_block_hash = unsafe_block_info.hash;
// process the execution payload
// TODO: needs testing
let (unsafe_block_info, import_outcome) =
match engine.handle_execution_payload(execution_payload, fcs).await {
Err(EngineDriverError::InvalidExecutionPayload) => (
Some(unsafe_block_info),
Some(Err(BlockImportError::Validation(BlockValidationError::InvalidBlock))),
),
Ok((PayloadStatusEnum::Valid, PayloadStatusEnum::Valid)) => (
Some(unsafe_block_info),
Some(Ok(BlockValidation::ValidBlock {
new_block: NewBlock {
block,
signature: signature.serialize_compact().to_vec().into(),
},
})),
),
_ => (None, None),
};
(unsafe_block_info, import_outcome.map(|result| BlockImportOutcome { peer, result }))
});

Upon block import result if success we update the fcs of the node manager.

/// Handles a block import outcome.
fn handle_block_import_outcome(
&mut self,
unsafe_block_info: Option<BlockInfo>,
outcome: Option<BlockImportOutcome>,
) {
trace!(target: "scroll::node::manager", ?outcome, "handling block import outcome");
// If we have an unsafe block info, update the forkchoice state.
if let Some(unsafe_block_info) = unsafe_block_info {
self.forkchoice_state.update_unsafe_block_info(unsafe_block_info);
}
// If we have an outcome, send it to the network manager.
if let Some(outcome) = outcome {
self.network.handle().block_import_outcome(outcome);
}
}

This works ok but I'm thinking we should introduce a State type that can be shared across async tasks (type State = Arc<Mutex<InnerState>>

@frisitano frisitano merged commit 65d9ef6 into main Feb 26, 2025
11 checks passed
greged93 pushed a commit that referenced this pull request Mar 12, 2025
* init bridge implementation

* extend bridge implementation

* refactor and clean up dependencies

* add .vscode to .gitignore

* lint

* spacing

* cleanup

* wrap inner block import in bridge

* add bridge integration test

* refactor and clean up

* add comments

* migrate shared dependencies to workspace

* feature propogation

* lints and feature fix

* remove scroll feature

* add scroll feature

* integrate reth upstream changes

* address PR feedback

* stash

* add systemd service file

* update systemd file

* update systemd file

* address comments

* rollup node manager

* merge main

* add Cargo.lock

* lint

* lint

* lint

* lint

* ci remove no-dev-deps

* fix serde dep - scroll-wire

* skip no_std on scroll-engine and scroll-rollup-node

* skip no_std on scroll-engine and scroll-rollup-node

* address comments

* lints

* cleanup

* tsets

* lint + refactor

* refactor engine api call ordering

* lint

* fix fcu

* lint

* remove syncing error return on new payload

* update forkchoice state after succesfully new payload

* update reth following chainspec fork id logic change

* PR feedback

* lints

* lints
frisitano added a commit that referenced this pull request Mar 19, 2025
* ci: add dprint

* feat: initial work + handle_latest_block

* Rollup node manager (#21)

* init bridge implementation

* extend bridge implementation

* refactor and clean up dependencies

* add .vscode to .gitignore

* lint

* spacing

* cleanup

* wrap inner block import in bridge

* add bridge integration test

* refactor and clean up

* add comments

* migrate shared dependencies to workspace

* feature propogation

* lints and feature fix

* remove scroll feature

* add scroll feature

* integrate reth upstream changes

* address PR feedback

* stash

* add systemd service file

* update systemd file

* update systemd file

* address comments

* rollup node manager

* merge main

* add Cargo.lock

* lint

* lint

* lint

* lint

* ci remove no-dev-deps

* fix serde dep - scroll-wire

* skip no_std on scroll-engine and scroll-rollup-node

* skip no_std on scroll-engine and scroll-rollup-node

* address comments

* lints

* cleanup

* tsets

* lint + refactor

* refactor engine api call ordering

* lint

* fix fcu

* lint

* remove syncing error return on new payload

* update forkchoice state after succesfully new payload

* update reth following chainspec fork id logic change

* PR feedback

* lints

* lints

* feat(watcher): reorg detection

* feat(primitives): add batch and l1 transaction

* feat(watcher): contracts and constants

* feat(watcher): imports

* feat(watcher): add error types

* feat(watcher): handle l1 messages and commit logs

* feat: handle batch commit and finalize

* test: handle_latest_block

* test: wip events decoding

* test: commit and finalize batch logs

* test: utils

* feat: tracing

* test: wip reorg integration

* test: reorg detection integration test

* test: gaps

* fix: lints

* feat: point deps to default reth branch

* fix: security issue on ring

* fix: bump alloy

* feat: move abi to separate crate

* chore: remove ArbitraryTxBuilder

* fix: answer comments

* fix: lints

* fix: no std issues

* fix: cargo features check

* feat: bounded vec

* fix: lints

---------

Co-authored-by: frisitano <35734660+frisitano@users.noreply.github.com>
frisitano added a commit that referenced this pull request Mar 19, 2025
* ci: add dprint

* feat: initial work + handle_latest_block

* Rollup node manager (#21)

* init bridge implementation

* extend bridge implementation

* refactor and clean up dependencies

* add .vscode to .gitignore

* lint

* spacing

* cleanup

* wrap inner block import in bridge

* add bridge integration test

* refactor and clean up

* add comments

* migrate shared dependencies to workspace

* feature propogation

* lints and feature fix

* remove scroll feature

* add scroll feature

* integrate reth upstream changes

* address PR feedback

* stash

* add systemd service file

* update systemd file

* update systemd file

* address comments

* rollup node manager

* merge main

* add Cargo.lock

* lint

* lint

* lint

* lint

* ci remove no-dev-deps

* fix serde dep - scroll-wire

* skip no_std on scroll-engine and scroll-rollup-node

* skip no_std on scroll-engine and scroll-rollup-node

* address comments

* lints

* cleanup

* tsets

* lint + refactor

* refactor engine api call ordering

* lint

* fix fcu

* lint

* remove syncing error return on new payload

* update forkchoice state after succesfully new payload

* update reth following chainspec fork id logic change

* PR feedback

* lints

* lints

* feat(watcher): reorg detection

* feat(primitives): add batch and l1 transaction

* feat(watcher): contracts and constants

* feat(watcher): imports

* feat(watcher): add error types

* feat(watcher): handle l1 messages and commit logs

* feat: handle batch commit and finalize

* test: handle_latest_block

* test: wip events decoding

* test: commit and finalize batch logs

* test: utils

* feat: tracing

* test: wip reorg integration

* test: reorg detection integration test

* test: gaps

* fix: lints

* feat: point deps to default reth branch

* fix: security issue on ring

* fix: bump alloy

* feat: move abi to separate crate

* chore: remove ArbitraryTxBuilder

* init database

* add database iterator

* add test

* update streaming implementation

* fix: answer comments

* fix: lints

* fix: no std issues

* fix: cargo features check

* extend and refactor database, migrations and batch primitive

* lint

* lint

* lint

* improve database implementation

* lint

* lint

* database transaction refactor

* lint:

* use GAT for DatabaseConnectionProvider trait

* refactor & lint

* refactor databse interface for insert_batch_input

* lint:

* lint

---------

Co-authored-by: Gregory Edison <gregory.edison1993@gmail.com>
frisitano added a commit that referenced this pull request Mar 19, 2025
* ci: add dprint

* feat: initial work + handle_latest_block

* Rollup node manager (#21)

* init bridge implementation

* extend bridge implementation

* refactor and clean up dependencies

* add .vscode to .gitignore

* lint

* spacing

* cleanup

* wrap inner block import in bridge

* add bridge integration test

* refactor and clean up

* add comments

* migrate shared dependencies to workspace

* feature propogation

* lints and feature fix

* remove scroll feature

* add scroll feature

* integrate reth upstream changes

* address PR feedback

* stash

* add systemd service file

* update systemd file

* update systemd file

* address comments

* rollup node manager

* merge main

* add Cargo.lock

* lint

* lint

* lint

* lint

* ci remove no-dev-deps

* fix serde dep - scroll-wire

* skip no_std on scroll-engine and scroll-rollup-node

* skip no_std on scroll-engine and scroll-rollup-node

* address comments

* lints

* cleanup

* tsets

* lint + refactor

* refactor engine api call ordering

* lint

* fix fcu

* lint

* remove syncing error return on new payload

* update forkchoice state after succesfully new payload

* update reth following chainspec fork id logic change

* PR feedback

* lints

* lints

* feat(watcher): reorg detection

* feat(primitives): add batch and l1 transaction

* feat(watcher): contracts and constants

* feat(watcher): imports

* feat(watcher): add error types

* feat(watcher): handle l1 messages and commit logs

* feat: handle batch commit and finalize

* test: handle_latest_block

* test: wip events decoding

* test: commit and finalize batch logs

* test: utils

* feat: tracing

* test: wip reorg integration

* test: reorg detection integration test

* test: gaps

* fix: lints

* feat: point deps to default reth branch

* fix: security issue on ring

* fix: bump alloy

* feat: move abi to separate crate

* chore: remove ArbitraryTxBuilder

* init database

* add database iterator

* add test

* update streaming implementation

* init indexer

* fix: answer comments

* fix: lints

* fix: no std issues

* fix: cargo features check

* extend indexer

* extend and refactor database, migrations and batch primitive

* lint

* lint

* lint

* improve database implementation

* lint

* lint

* database transaction refactor

* lint:

* use GAT for DatabaseConnectionProvider trait

* refactor & lint

* merge database and refactor

* clippy fix

* skip rolllup-node-indexer no_std

* refactor databse interface for insert_batch_input

* refactor

* lint

* refactor indexer

* refactor indexer

* lint:

* lint

---------

Co-authored-by: Gregory Edison <gregory.edison1993@gmail.com>
frisitano added a commit that referenced this pull request Mar 26, 2025
* ci: add dprint

* feat: initial work + handle_latest_block

* Rollup node manager (#21)

* init bridge implementation

* extend bridge implementation

* refactor and clean up dependencies

* add .vscode to .gitignore

* lint

* spacing

* cleanup

* wrap inner block import in bridge

* add bridge integration test

* refactor and clean up

* add comments

* migrate shared dependencies to workspace

* feature propogation

* lints and feature fix

* remove scroll feature

* add scroll feature

* integrate reth upstream changes

* address PR feedback

* stash

* add systemd service file

* update systemd file

* update systemd file

* address comments

* rollup node manager

* merge main

* add Cargo.lock

* lint

* lint

* lint

* lint

* ci remove no-dev-deps

* fix serde dep - scroll-wire

* skip no_std on scroll-engine and scroll-rollup-node

* skip no_std on scroll-engine and scroll-rollup-node

* address comments

* lints

* cleanup

* tsets

* lint + refactor

* refactor engine api call ordering

* lint

* fix fcu

* lint

* remove syncing error return on new payload

* update forkchoice state after succesfully new payload

* update reth following chainspec fork id logic change

* PR feedback

* lints

* lints

* feat(watcher): reorg detection

* feat(primitives): add batch and l1 transaction

* feat(watcher): contracts and constants

* feat(watcher): imports

* feat(watcher): add error types

* feat(watcher): handle l1 messages and commit logs

* feat: handle batch commit and finalize

* test: handle_latest_block

* test: wip events decoding

* test: commit and finalize batch logs

* test: utils

* feat: tracing

* test: wip reorg integration

* test: reorg detection integration test

* test: gaps

* fix: lints

* feat: point deps to default reth branch

* fix: security issue on ring

* fix: bump alloy

* feat: move abi to separate crate

* chore: remove ArbitraryTxBuilder

* init database

* add database iterator

* add test

* update streaming implementation

* init indexer

* fix: answer comments

* fix: lints

* fix: no std issues

* fix: cargo features check

* extend indexer

* integrate and refactor

* extend and refactor database, migrations and batch primitive

* lint

* lint

* lint

* improve database implementation

* lint

* lint

* database transaction refactor

* lint:

* use GAT for DatabaseConnectionProvider trait

* refactor & lint

* merge database and refactor

* clippy fix

* skip rolllup-node-indexer no_std

* refactor databse interface for insert_batch_input

* lint and refactor

* refactor and lint

* lint

* refactor

* lint

* merge indexer refactor

* refactor indexer

* refactor indexer

* chore: integrate upstream

* fix: fix no_std for scroll-wire

* chore: point reth dep and rename rollup-node binary

* lint

* fix: re-add scroll-codec to riscV skip lint

---------

Co-authored-by: Gregory Edison <gregory.edison1993@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants