test red 201 b #290
Merged
test red 201 b #290
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tting the previous records
…w fields to db. Expose new functions in shardus to allow for adding network transactions and verifiers.Minor cleanup of unused logs
…lidation for this. Do not add tx that has already been added to txadd
…to txList and implement reset function
…nd verify in handler
* move module functions around * organize ServiceQueue module around existing module structure * add syncTxListFromArchiver func * chore: lint ServiceQueue * chore: move variables above routes * switch to using random archiver for syncing tx list * add syncing tx list step to restore callback * remove unused import * add await to sync tx list func call ;)
code clean up
chore: Add debug flag for NTP bogus offset decrements chore: Update time synchronization error messages and comment debug endpoint chore: Remove commented debug endpoint for NTP bogus offset decrements
Bumps the npm_and_yarn group with 4 updates in the / directory: [express](https://github.com/expressjs/express), [socket.io](https://github.com/socketio/socket.io), [braces](https://github.com/micromatch/braces) and [tar](https://github.com/isaacs/node-tar). Updates `express` from 4.16.4 to 4.19.2 - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.16.4...4.19.2) Updates `socket.io` from 2.4.1 to 2.5.0 - [Release notes](https://github.com/socketio/socket.io/releases) - [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md) - [Commits](socketio/socket.io@2.4.1...2.5.0) Updates `braces` from 3.0.2 to 3.0.3 - [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md) - [Commits](micromatch/braces@3.0.2...3.0.3) Updates `tar` from 6.1.15 to 6.2.1 - [Release notes](https://github.com/isaacs/node-tar/releases) - [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md) - [Commits](isaacs/node-tar@v6.1.15...v6.2.1) --- updated-dependencies: - dependency-name: express dependency-type: direct:production dependency-group: npm_and_yarn - dependency-name: socket.io dependency-type: direct:production dependency-group: npm_and_yarn - dependency-name: braces dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: tar dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com>
fix: add logging to 404 case in dumpTree
remove comments
…tion on fetchLatestCycle
Make safetyNum optional
…ameters and payload) (#255) * feat: adding replay attack protection for debug endpoints' payloads * chore : formatting
mhanson-github
added a commit
that referenced
this pull request
Sep 11, 2024
* init commit for serviceQueue * add serviceQueue to storage model * more storage updates for serviceQueue * fix: update to use serviceQueue types * Add activeCycle to node record. Fix bug where serviceQueue was not getting the previous records * Add initial commit for processing flow * feat: add basic functions for generic ServiceQueue add/remove * fix: crypto import * fix: remove route * Fix cycle record to include new txadd,remove, listhash fields. Add new fields to db. Expose new functions in shardus to allow for adding network transactions and verifiers.Minor cleanup of unused logs * fix init in ServiceQueue to register gossip instead of internal routes * Define NetworkTransaction to enforce data having an id. Additional validation for this. Do not add tx that has already been added to txadd * remove NetworkTransaction type in favor of OpaqueTransaction. Add tx to txList and implement reset function * basic remove flow * fix same tx getting processed twice in a row * change datatype of txlist to array and use sorting for insert * various small fixes * send txAdd and txRemove gossip in sendRequests. SIgn gossip payload and verify in handler * send gossip in sendRequests. add txadd and txremove to getTxs. * remove unused lines. signing is done elsewhere * feat: add node to node sync for txList * fix: tx-list-hash returning wrong hash * add additionalData to ShardusEvent to support custom data for network transactions * add additional data to try-network-transaction event * add activeCycle when node deactivates * put logging behind flag * fixes for applyVerifier * Red 203 (#227) * move module functions around * organize ServiceQueue module around existing module structure * add syncTxListFromArchiver func * chore: lint ServiceQueue * chore: move variables above routes * switch to using random archiver for syncing tx list * add syncing tx list step to restore callback * remove unused import * add await to sync tx list func call ;) * fixes + added subQueue key * refactoring processNetworkTransactions * put txHash not found in txList on remove behind log flag * addnetworktx fixes * fix minor bugs * remove sign from tx data when adding to txlist * move processnetworktransactions to q3 * use hash without sig * chore: lint + prettify * move changes to txList to parseRecord * refactor sortedInsert to accept a list as input * set txListHash after applying txadd and txremove to txListCopy * use NetworkTxEntry type * use SeriveQueueTypes.Txs type * fix: pass list to all calls of sortedInsert in ServiceQueue * update sortedInsert calss * add back sign to destructuring txData * fix: change where we push to txAdd and txRemove * OOS fixes for serviceQueue * add debug counter endpiont * more OOS fixes * fix: dont gossip multiple times * update gossip early return check for add and remove * fix: potential cycle OOS case - push to txAdd and txRemove in sendRequests and gossip handlers only * fix: sort txadd and txremove before adding to cycle record * fix: various cycle OOS fixes * pass whole txEntry not just txData to verifiers * remove log * reduce unnecessary gossip * adjust how we sync txList from archiver in restart mode * update logs in syncTxListFromArchiver func * add all active nodes to txAdd in event of shutdown * update byPubKey usage + lint * register dapp interface func getDeactivatedTxData * ServiceQueue: fixes and improvements for shutdown case * fix(WIP): shutdown causing stuck txs * move processing check to top of function for early return * show whole txlist entries on count endpoint * remove getDeactivatedTxData as its not needed * add missing field in cycle model * update cycle column * archiversAtShutdown should be json and allowed to be null * early return on shutdown to not emit activated/deactivated events * check record for mode * add cycle mode check as well for early return * add cycle mode check as well for early return * fix counts * hold proposals until processing mode before attempting * only process txs in q3 * feat: schema names literal string be replaced with an enum wip: setup ajv for binary_repair_oos_account feat: adding ajv for following endpoint (commit body) Added Ajvs for endpoints - binary_repair_oos_account, binary_request_receipt_for_tx, binary_request_state_for_tx, binary_get_applied_vote, binary_lost_report fix: added unit test and fixes bug Added Unit tests for ajv(s) that were added in previous commit and implement fixes for bug that has been uncover by the unit test. This commit is as part of internal Hardening project - Kaung(GOLD team) feat: Add missing comma in TypeIdentifierEnum The commit adds a missing comma in the `TypeIdentifierEnum` enum, specifically in the `cRepairOOSAccountsReq` value. This fix ensures that the enum is correctly defined and avoids any syntax errors. feat: Remove console.log statements and optimize deserialization The code changes remove unnecessary console.log statements and optimize the deserialization process in the `deserializeRequestStateForTxResp` function. This improves the performance and readability of the code. AJV validation and unit tests for RequestStateForTxPost Refactor code for endpoint hardening binary_broadcast_state ajv hardening and unit tests remove broadcast state ajv check in handler GetAccountDataResp Hardening and Unit Tests Using enum for reference instead of string fixed request payload validation GetAccountDataReq Unit Tests fix: Update verifyAddressRange function name in AccountSync apoptosis ajv hardening and unit tests remove apoptosis ajv check in handler chore: Remove unused RepairMissingAccountsReq schema and related code chore: reformatted using prettier refactor: Update AJV schema references to use enum instead of string AJV Enum Changes and Code Cleaning GetAccountData Enum changes for AJV GetTxTimestampResp Signature Object Detection Fixed serialization testing logic in GetTxTimestampResp fix: apop endpoint name Co-authored-by: Sonali Thakur <sonali@shardeum.org> Co-authored-by: Kaung Myattthu <2962000kaungmyatthu@gmail.com> Co-authored-by: sabdulazeem01 <sabdulazeem01@gmail.com> Co-authored-by: Jintu Das <jintu@shardeum.org> fix(ut): fixed applied receipt 2 ut * feat: Add AJV schema for GetCachedAppDataReq and GetCachedAppDataResp (#218) Added AJV schemas for the GetCachedAppDataReq and GetCachedAppDataResp types. This ensures that the request and response payloads for the GetCachedAppData endpoint are properly validated. The schemas have been added to the AJV schema registry and are now available for validation. * AJV validation and unit tests for CompareCert (#217) * chore: fix AjvSchemaEnum for GetAccountDataByList Req and Resp (#216) * GOLD-203: AJV addition and verification data check: binary_get_trie_account_hashes (#221) * Added ajv verification and unit tests for binary_get_trie_account_hashes * Added enums * ajv schema for binary_get_account_data_with_queue_hints (#214) * feat: Add SpreadAppliedVoteHashReq Ajv schema and unit tests (#213) Added the SpreadAppliedVoteHashReq schema and its initialization in the AJV schemas. This schema defines the structure for spreading applied vote hash requests, including the transaction ID, vote hash, and signature. This addition enhances the validation and serialization capabilities of the application justified by the unit tests. * AJV addition and verification data check: binary_get_globalaccountreport (#215) * feat: migrate ajv broadcastFinalStateReq and unit test (#212) * BLUE-175 Add AJV validation to syncTrieHashes endpoint (#210) * Add AJV validation to syncTrieHashes endpoint * chore: audit gold-134 to confirm with naming patterns undo out of scope changes * Add unit test cases for SyncTrieHashesReq --------- Co-authored-by: Kaung Myat Thu <2962000kaungmyatthu@gmail.com> * GetAccountQueueCount ajv hardening and unit tests (#155) * GOLD-130: AJV addition and verification data check: binary_make_receipt (#154) * AJV validation and unit tests for MakeReceipt * Use new enum structure * Add AJV validation and unit test cases for GetTrieHashes endpoint (#211) Add AJV validation and unit test cases for cachedAppData and sendCachedAppDataReq (#225) * fix(enum): migration to use AJV enum * fix(ut): updated unit tests for send cached app data req * fix(rm): rmoved unused test code * Add some debug endpoints for transaction time testing. * fix: add logging when setting faked time offset * Update logging * add queueLengthAll, tx age buckets to report * RED-13 cycleTX for unjoin wip + dev logs + comments fix: handle case of node unjoining the same cycle it is selected add edge case node to apop instead fo removed + add optimization comment feat: get unjoin cycle txs working chore: code cleanup + remove dev logs chore: adjust a log's logflag level * Update function to handle isOrig & !payload Add payload validation in gossip-active handler Add validation and error handling for gossip handlers in Active and Join Add type validation and signature verification to lostArchiverUpGossip Remove forwarding of gossip after quarter 2 in Active.ts Add input validation and error handling for joinarchiver and lostArchiverDownGossip Add check to not gossip forwarding after quarter 2 at end remove todo refactor validation and logging in Archivers and LostArchivers routes Remove redundant forwarding of gossip after quarter 2 Refactor sign object validation in gossipSyncFinishedRoute and gossipStandbyRefresh add back sender and orig check in gossipStandbyRefresh and lostArchiverDownGossip handlers remove comment Refactor validation and add logging for rejected gossip messages Add payload stucture and type validation in gossipSyncStartedRoute Fix check for quarter 1 and 2 for sync-finished rejection in gossipSyncFinishedRoute Remove extra import of nestedCountersInstance Add ReceiptsBundleByInterval enum to Archivers.ts Refactor archiver data subscriptions and add back API endpoint for checking joined archivers Add back type assertion in routes.ts Add back logs from dev in join request validation remove signature verification using nodelist Update appJoinData type in gossipJoinRoute and gossipValidJoinRequests Add GossipValidation module for validating gossip payloads Add GossipValidation to Active, and Archivers routes Update GossipValidation function names Update GossipValidation to include current cycle in rejection message Refactor comments for clarity Refactor checkGossipPayload to include verifyOriginalSenderAndQuarter and remove OOS logging Remove line Fix sync-started failure reason count in gossipSyncStartedRoute Remove quarter check in unjoin request gossip handler Move validation logic for original transactions after quarter 1 & 2 checking logic Refactor GossipValidation.ts to simplify code and improve error logging sign joinRequest payload and require signer in `checkGossipPayload` otherwise return false Refactor joinRequest payload signing and validation Refactor joinRequest payload validation and signing Refactor validated joinRequest payload signing and validation Add and remove comments and add JSDoc comment for checkGossipPayload Add count event for invalid payload.joinRequest in gossipValidJoinRequests Remove commneted out code and add new line to end of file Add checkGossipPayload to gossipUnjoinRequests handler due to red-13 PR involving quarters * error handling for all external endpoints * update endpoint-exception counter message * initial adjustments to join cycle txs to use getTxs * update standbyadd typing and add standbyremove in cycle txs * add lostAfterSelection cycle txs * move unjoin type to shardus types * Revert "add lostAfterSelection cycle txs" This reverts commit 8aa0e8c. * Fix missed adding syncRequest in newSyncStarted * Update what node list we check that a requested cycle tx node is on * fix: cycle tx merge issues * fix: handle unjoin gossip in GossipValidation properly * fix: cycle OOS coming from maxSyncTime * use latest shardus types * Bump version to 2.12.30-71 * adjust logFlag and interval for memory logging adjust interval for memory logging to 300s * remove unused non binary internal endpoints block non binary internal point registration remove request_receipt_for_tx_old unblock non binary internal point registration * error handling for all external endpoints update endpoint-exception counter message * feat(counters): added gossip type to gossip handler exception counter * add catch for apoptosize endpoint code clean up * chore: Update time synchronization logic chore: Add debug flag for NTP bogus offset decrements chore: Update time synchronization error messages and comment debug endpoint chore: Remove commented debug endpoint for NTP bogus offset decrements * feat: Fix rare race condition serialisation in getTxTimestamp * Temporarily change a timing value. * Bump the npm_and_yarn group across 1 directory with 4 updates Bumps the npm_and_yarn group with 4 updates in the / directory: [express](https://github.com/expressjs/express), [socket.io](https://github.com/socketio/socket.io), [braces](https://github.com/micromatch/braces) and [tar](https://github.com/isaacs/node-tar). Updates `express` from 4.16.4 to 4.19.2 - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.16.4...4.19.2) Updates `socket.io` from 2.4.1 to 2.5.0 - [Release notes](https://github.com/socketio/socket.io/releases) - [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md) - [Commits](socketio/socket.io@2.4.1...2.5.0) Updates `braces` from 3.0.2 to 3.0.3 - [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md) - [Commits](micromatch/braces@3.0.2...3.0.3) Updates `tar` from 6.1.15 to 6.2.1 - [Release notes](https://github.com/isaacs/node-tar/releases) - [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md) - [Commits](isaacs/node-tar@v6.1.15...v6.2.1) --- updated-dependencies: - dependency-name: express dependency-type: direct:production dependency-group: npm_and_yarn - dependency-name: socket.io dependency-type: direct:production dependency-group: npm_and_yarn - dependency-name: braces dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: tar dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> * fix: handle undefined case for `debug-patcher-dumpTree-partial` fix: add logging to 404 case in dumpTree * put log behind logflags * fix: convert the cache datastruct to be a map prevents proto pollute * pri-55. work from Sonali, had to extract from a merge commit * Normalize paths for trie and comparison in debug.ts remove comments * SEC-407: Change precrack to run only when the tx is pulled from nonceQueue * use latest shardus types and net. fix mege typo * Bump version to 2.13.0 * add acceptedTx in the debugQueueInfo and remove unnecessary POQLS related info * Temp fix for itn2 syncing nodes crashing due to the failure in validation on fetchLatestCycle * add exception handling * Added more logs for BLUE-213; * Bump version to 2.13.1-0 * Make safetyMode an optional param to validate Make safetyNum optional * PRI-53 : Harden debug middleware against replay attacks (sign the parameters and payload) (#255) * feat: adding replay attack protection for debug endpoints' payloads * chore : formatting * remove processing check from sendRequests * add init rewards for nodes that rotated in during shutdown * expose simplified fastIsPick to server * add `cycleSinceActivated` to txDebug queueEntry --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: urnotsam <samwsweet@gmail.com> Co-authored-by: Sam Sweet <sam.w.sweet@gmail.com> Co-authored-by: CombatPug <pe.ladar@icloud.com> Co-authored-by: Ahmed Iqbal <64235582+ahmxdiqbal@users.noreply.github.com> Co-authored-by: Ahmed <iaahmed40@gmail.com> Co-authored-by: Arham Jain <arham.nec@gmail.com> Co-authored-by: Arham <86415475+arhamj@users.noreply.github.com> Co-authored-by: JINTU KUMAR DAS <jintu@shardeum.org> Co-authored-by: Abdul Azeem Shaik <sabdulazeem01@gmail.com> Co-authored-by: Sonali Thakur <sonali@shardeum.org> Co-authored-by: Kaung Myat Thu <2962000kaungmyatthu@gmail.com> Co-authored-by: tanuj-shardeum <153592530+tanuj-shardeum@users.noreply.github.com> Co-authored-by: mhanson <3117293+mhanson-github@users.noreply.github.com> Co-authored-by: Zachary Belford <belfordz66@gmail.com> Co-authored-by: dnlbui <dnlbui@proton.me> Co-authored-by: Sonali Thakur <sonalithakur1199@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thura Moe Myint <mgthuramoemyint@gmail.com> Co-authored-by: Thant Sin Toe <mr.thantsintoe@gmail.com> Co-authored-by: jairajdev <jairajranamagar.dev@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.