Apparent codec error in validate_transaction

[miloskriz]

Dear team, hello,

We just saw a burst of panic messages in all our kusama validators:

2024-04-19 22:08:16 💤 Idle (22 peers), best: #22812271 (0x6bc7…33b2), finalized #22812268 (0xff45…fa75), ⬇ 2.2MiB/s ⬆ 2.8MiB/s
2024-04-19 22:08:18 ✨ Imported #22812272 (0x5ef2…a636)
2024-04-19 22:08:21 💤 Idle (23 peers), best: #22812272 (0x5ef2…a636), finalized #22812270 (0xa884…85f1), ⬇ 2.2MiB/s ⬆ 3.5MiB/s
2024-04-19 22:08:22 panicked at /build/relay/kusama/src/lib.rs:2082:1:
Bad input data provided to validate_transaction: Codec error
2024-04-19 22:08:22 panicked at /build/relay/kusama/src/lib.rs:2082:1:
Bad input data provided to validate_transaction: Codec error
2024-04-19 22:08:22 panicked at /build/relay/kusama/src/lib.rs:2082:1:
Bad input data provided to validate_transaction: Codec error

(... dozens of the same ...)

2024-04-19 22:08:22 panicked at /build/relay/kusama/src/lib.rs:2082:1:
Bad input data provided to validate_transaction: Codec error
2024-04-19 22:08:24 ✨ Imported #22812273 (0xb62a…7260)
2024-04-19 22:08:24 ♻️  Reorg on #22812273,0xb62a…7260 to #22812273,0x8c00…28fd, common ancestor #22812272,0x5ef2…a636
2024-04-19 22:08:24 ✨ Imported #22812273 (0x8c00…28fd)
2024-04-19 22:08:26 💤 Idle (24 peers), best: #22812273 (0x8c00…28fd), finalized #22812270 (0xa884…85f1), ⬇ 2.0MiB/s ⬆ 2.8MiB/s
2024-04-19 22:08:30 ✨ Imported #22812274 (0x39aa…f237)

We don't know if related but there seems to be a historical case of the same documented here

Concurrently, my node experienced some struggles during its paravalidation slot?

Please advise!

Many thanks!!

Milos

[bkchr]

Concurrently, my node experienced some struggles during its paravalidation slot?

Yeah these validate_transaction issues should not be related. What struggles did you have?

[miloskriz]

Dear Bastian, hello!

Thanks for your clarification!...

You are correct, my issue was unrelated but it is really interesting (concerning?) as it was apparently caused by some odd problems of polkajs to decode my previous BEEFY keys (generated and first applied 10 Nov 2023):

[
  {
    name: "keys",
    type: "staging_kusama_runtime:SessionKeys",
    type_name: "Keys",
    value: {
      authority_discovery: "0xc4a1777654cb2a192b6b556c21bc9041b7ccdda01a1cc906cf6c614347572e7c",
      babe: "0xf6127b05312894d8670489109553b710094f1874bd8979cd2304ccd17699416c",
      beefy: "0x032016e2a5c0b2043d8c823817c932a3ddb8e2cfa7d517c4b2adc756dfac74644f",
      grandpa: "0x12a40e1893521aac001eff367c5fbb0eef42d44028757fbc25f8b9c00b7eacda",
      im_online: "0x1a53ed212b86748dacef04e3d093d7bed3dd1470550a01e1f25279a044a25524",
      para_assignment: "0xd2583527db26df18c1add3915644c3caf6e9573dbed4e3279ec9d65b646b5104",
      para_validator: "0x80f5898ad5a49004f44bf1e91d6da74848c428a1c103e2a01e78e27323a9f21c"
    }
  },
  {
    name: "proof",
    type: "Vec<U8>",
    type_name: "Bytes",
    value: ""
  }
]

After doing maintenance last week (and moving validator to and from backup machine) the very same session keys were decoded as:

[
  {
    name: "keys",
    type: "staging_kusama_runtime:SessionKeys",
    type_name: "Keys",
    value: {
      authority_discovery: "0xd2583527db26df18c1add3915644c3caf6e9573dbed4e3279ec9d65b646b5104",
      babe: "0xf6127b05312894d8670489109553b710094f1874bd8979cd2304ccd17699416c",
      beefy: "0xc4a1777654cb2a192b6b556c21bc9041b7ccdda01a1cc906cf6c614347572e7c03",
      grandpa: "0x12a40e1893521aac001eff367c5fbb0eef42d44028757fbc25f8b9c00b7eacda",
      para_assignment: "0x80f5898ad5a49004f44bf1e91d6da74848c428a1c103e2a01e78e27323a9f21c",
      para_validator: "0x1a53ed212b86748dacef04e3d093d7bed3dd1470550a01e1f25279a044a25524"
    }
  },
  {
    name: "proof",
    type: "Vec<U8>",
    type_name: "Bytes",
    value: ""
  }
]

You can see the grandpa and babe keys were ok (thus the reason I was able to produce blocks) but the rest were a mess (swapped keys between authority_discovery and para_assignment and para_validator and im_online, wrong beefy, etc.)

I have since rotated Keys again and everything is back on track 😃, but it was a painful problem to diagnose😫

Any thoughs?

Many thanks!

Milos

[bkchr]

Not sure exactly what you have done there. If you called rotate_keys it will give you new keys for every sub key. Do you call set_keys "manually"?

[miloskriz]

Hi Bastian!...

to do some maintenance of main node I used the menu option "Change session keys" from the polka{.js] in Network->Staking->Accounts page.

I then provided the very same hex value (0x...) of the keyset from my spare validator, this value was saved from last rotate_keys performed 10 Nov 2023, unfortunately they got decoded wrong.

My suspicion is that, as the im_online is now obsolete and its key is no longer parsed, the hex value of the original keyset decoded the rest of the keys corruptly.

hope this helps?.. I confirm that I had to perform rotate_keys again to generate a new hex with beefy, but without im_online and everything is back to normal.

Many thanks!!

Best regards

Milos

[bkchr]

My suspicion is that, as the im_online is now obsolete and its key is no longer parsed, the hex value of the original keyset decoded the rest of the keys corruptly.

Yeah that is the issue. im_online was removed and should not be passed there anymore. You could maybe open an issue in polkadot-js to print an error or similar if you pass to many bytes and it doesn't use all of them for decoding.

Given that, I think we can close this issue.