Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snapshots: SnapshotPath.step_range #2419

Merged
merged 16 commits into from
Oct 15, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SnapshotSync::seed_frozen_local_snapshots: don't use max_block_number…
…, remove compute_max_block
battlmonstr committed Oct 14, 2024
commit 735bc218fed506333232eceb774c7f7cbfff99ce
28 changes: 8 additions & 20 deletions silkworm/db/datastore/snapshots/config.cpp
Original file line number Diff line number Diff line change
@@ -16,6 +16,7 @@

#include "config.hpp"

#include <ranges>
#include <span>

#include <silkworm/core/common/small_map.hpp>
@@ -58,25 +59,6 @@ Config Config::lookup_known_config(
return Config{std::move(entries)};
}

Config::Config(PreverifiedList entries)
: entries_(std::move(entries)),
max_block_number_(compute_max_block(entries_)) {
}

BlockNum Config::compute_max_block(const PreverifiedList& entries) {
BlockNum max_block{0};
for (const auto& entry : entries) {
const auto snapshot_path = SnapshotPath::parse(std::filesystem::path{entry.file_name});
if (!snapshot_path) continue;
if (snapshot_path->extension() != kSegmentExtension) continue;
if (snapshot_path->type() != SnapshotType::headers) continue;
if (snapshot_path->block_range().end > max_block) {
max_block = snapshot_path->block_range().end;
}
}
return max_block > 0 ? max_block - 1 : 0;
}

PreverifiedList Config::remove_unsupported_entries(const PreverifiedList& entries) {
static constexpr std::array kUnsupportedSnapshotNameTokens = {
"/"sv,
@@ -89,7 +71,7 @@ PreverifiedList Config::remove_unsupported_entries(const PreverifiedList& entrie

// erase file names containing any of the unsupported tokens
std::erase_if(results, [&](const Entry& entry) {
return std::any_of(kUnsupportedSnapshotNameTokens.begin(), kUnsupportedSnapshotNameTokens.end(), [&entry](const auto& token) {
return std::ranges::any_of(kUnsupportedSnapshotNameTokens, [&entry](std::string_view token) {
// NOLINTNEXTLINE(abseil-string-find-str-contains)
return entry.file_name.find(token) != std::string_view::npos;
});
@@ -98,4 +80,10 @@ PreverifiedList Config::remove_unsupported_entries(const PreverifiedList& entrie
return results;
}

bool Config::contains_file_name(std::string_view file_name) const {
return std::ranges::any_of(entries_, [&](const Entry& entry) {
return entry.file_name == file_name;
});
}

} // namespace silkworm::snapshots
9 changes: 4 additions & 5 deletions silkworm/db/datastore/snapshots/config.hpp
Original file line number Diff line number Diff line change
@@ -35,17 +35,16 @@ class Config {
ChainId chain_id,
std::optional<std::function<bool(std::string_view file_name)>> include_filter_opt = std::nullopt);

explicit Config(PreverifiedList entries);
explicit Config(PreverifiedList entries)
: entries_(std::move(entries)) {}

[[nodiscard]] const PreverifiedList& preverified_snapshots() const { return entries_; }
[[nodiscard]] BlockNum max_block_number() const { return max_block_number_; }
const PreverifiedList& preverified_snapshots() const { return entries_; }
bool contains_file_name(std::string_view file_name) const;

private:
static BlockNum compute_max_block(const PreverifiedList& entries);
static PreverifiedList remove_unsupported_entries(const PreverifiedList& entries);

PreverifiedList entries_;
BlockNum max_block_number_;
};

} // namespace silkworm::snapshots
6 changes: 0 additions & 6 deletions silkworm/db/datastore/snapshots/snapshot_path.hpp
Original file line number Diff line number Diff line change
@@ -22,8 +22,6 @@
#include <string>
#include <vector>

#include <silkworm/core/common/base.hpp>

#include "snapshot_type.hpp"
#include "step.hpp"

@@ -51,10 +49,6 @@ class SnapshotPath {
std::string extension() const { return path_.extension().string(); }
uint8_t version() const { return version_; }
StepRange step_range() const { return step_range_; }
// TODO: remove
BlockNumRange block_range() const {
return step_range().to_block_num_range();
}
SnapshotType type() const { return type_; }
std::string type_string() const;
bool exists() const { return std::filesystem::exists(path_); }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why adding this helper method and removing similar ones for last_write_time and index_file_for_type or snapshot_path_for_type?
I'm fine with this exists helper, but I suggest to keep the existing ones as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing path_for_type is not related to steps indeed, but as I wanted to keep SnapshotPath as clean as possible, somehow I couldn't resist doing it in this PR :) I agree that the path_for_type is nice, and I will introduce it later in a different schema-related class and replace direct related_path calls to a higher level API.

The idea is that in the future SnapshotPath will be just a container for meta-data encoded in the filename + the full path. In order to get SnapshotType (and things like index_file_for_type) the code will need to consult "schema" - a new decoupled object. So instead of deducing a schema element from path, we'll normally do the other way around: deduce path(s) from schema. The few cases where we need to go from path to schema will be a new method like schema.type_of_path(path). Removing path_for_type is a preparation for this change.

4 changes: 3 additions & 1 deletion silkworm/db/snapshot_sync.cpp
Original file line number Diff line number Diff line change
@@ -275,7 +275,9 @@ void SnapshotSync::seed_frozen_local_snapshots() {
for (auto& bundle_ptr : repository_.view_bundles()) {
auto& bundle = *bundle_ptr;
bool is_frozen = bundle.block_range().size() >= kMaxMergerSnapshotSize;
bool is_preverified = bundle.block_range().end <= snapshots_config_.max_block_number() + 1;
const auto first_snapshot = bundle.snapshots()[0];
// assume that if one snapshot in the bundle is preverified, then all of them are
bool is_preverified = snapshots_config_.contains_file_name(first_snapshot.get().path().filename());
if (is_frozen && !is_preverified) {
seed_bundle(bundle);
}