Capture SP task dumps in support bundles #8177

wfchandler · 2025-05-16T14:24:51Z

Update the support bundle collector to capture task dumps from the SPs.

wfchandler · 2025-05-20T20:49:40Z

Tested on Dublin:

# Create a task dump
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -a /tmp/build-gimlet-image-e.zip --ip fe80::aa40:25ff:fe04:604%gimlet15 dump --task control_plane_agent
humility: connecting to fe80::aa40:25ff:fe04:604%51
humility: using UDP dump agent
humility: pulled 10.12KB in 0 seconds
humility: dumping to "hubris.core.control_plane_agent.0"
humility: dumped 661.38KB in 0 seconds

# Confirm the dump is present on the SP
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -a /tmp/build-gimlet-image-e.zip --ip fe80::aa40:25ff:fe04:604%gimlet15 dump -l
humility: connecting to fe80::aa40:25ff:fe04:604%51
humility: using UDP dump agent
AREA TASK                  TIME       SIZE
   0 control_plane_agent   12942213   10368

# Contents of original task dump
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -d hubris.core.control_plane_agent.0 tasks
humility: attached to dump
system time = 12942213
ID TASK                       GEN PRI STATE    
19 control_plane_agent          0   7 recv, notif: usart-irq(irq37) socket timer

# Create a support bundle
root@oxz_switch0:/tmp# omdb -w nexus sb create
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:103::4]:12221
created support bundle: 39bff2f6-15c2-4f6e-93ba-c832380816e6

# Download the dehydrated task dump from the bundle
root@oxz_switch0:/tmp# omdb nexus sb get-file 39bff2f6-15c2-4f6e-93ba-c832380816e6 sp_task_dumps/sled_15/dump-0.zip > /tmp/dump-0.zip

# Rehydrate the dump from the archive
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless --archive /tmp/build-gimlet-image-e.zip hydrate dump-0.zip 
humility: read dehydrated crash dump
humility:   task index: 19
humility:   crash time: 12942213
humility:   archive id: [d5, 3a, 8f, a4, c9, b5, 6f, 91]
humility:   board:      gimlet-e
humility:   git commit: 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab
humility:   version:    1.0.37
humility:   2 memory regions:
humility:     0x24001108: 168 bytes
humility:     0x24030000: 32768 bytes
humility: dumping to "hubris.core.control_plane_agent.1"
humility: dumped 661.38KB in 0 seconds

# Confirm we have the same tasks
root@oxz_switch0:/tmp# /tmp/humility-0.12.0-probless -d hubris.core.control_plane_agent.1 tasks
humility: attached to dump
system time = 12942213
ID TASK                       GEN PRI STATE    
19 control_plane_agent          0   7 recv, notif: usart-irq(irq37) socket timer

# Rehydrated file matches the original
root@oxz_switch0:/tmp# shasum hubris.core.control_plane_agent.*
00a066ce8b911b09579648847650b096dd594831  hubris.core.control_plane_agent.0
00a066ce8b911b09579648847650b096dd594831  hubris.core.control_plane_agent.1

wfchandler · 2025-05-20T20:54:07Z

nexus/src/app/background/init.rs

@@ -505,6 +505,7 @@ impl BackgroundTasksInitializer {
            task_impl: Box::new(
                support_bundle_collector::SupportBundleCollector::new(
                    datastore.clone(),
+                    resolver.clone(),


Maybe this isn't the best approach, given the conversation in the control plane huddle today. On the other hand, we're very unlikely to be collecting bundles fast enough to have a real impact.

I'm fine to punt on this until we sort out the progenitor integration with qorb more holistically

smklein · 2025-05-20T21:00:31Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        tokio::fs::create_dir_all(&sp_dumps_dir).await.with_context(|| {
+            format!("failed to create SP task dump directory {sp_dumps_dir}")
+        })?;
+        let sp_dumps_fut =


We should strongly consider modifying SupportBundleCollectionReport in nexus/types/src/internal_api/background.rs to identify whether or not collection of the SP dumps was successful or not.

smklein · 2025-05-20T21:01:08Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        let sp_dumps_dir = dir.path().join("sp_task_dumps");
+        tokio::fs::create_dir_all(&sp_dumps_dir).await.with_context(|| {
+            format!("failed to create SP task dump directory {sp_dumps_dir}")
+        })?;


This looks replicated from above? We should define this variable / create_dir_all in one spot, perhaps?

smklein · 2025-05-20T21:01:28Z

nexus/src/app/background/init.rs

@@ -505,6 +505,7 @@ impl BackgroundTasksInitializer {
            task_impl: Box::new(
                support_bundle_collector::SupportBundleCollector::new(
                    datastore.clone(),
+                    resolver.clone(),


I'm fine to punt on this until we sort out the progenitor integration with qorb more holistically

smklein · 2025-05-20T21:03:02Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+        .context("failed to get list of SPs from MGS")?
+        .into_inner();
+
+    let mut futures = futures::stream::iter(all_sps.into_iter())


I recommend checking out parallel-task-set, which we added relatively recently, to perform this task-saving operation in parallel.

(It should be quite similar to using buffer_unordered, but it'll actually spawn tokio tasks for each SP - but we can still set a limit on the maximum amount of parallelism)

smklein · 2025-05-20T21:06:19Z

nexus/src/app/background/tasks/support_bundle_collector.rs

+    sp_dumps_dir: &Utf8Path,
+) -> anyhow::Result<()> {
+    let dump_count = mgs_client
+        .sp_task_dump_count(sp.type_, sp.slot)


When / how are these task dumps deleted? Just trying to understand if there's a TOCTTOU issue between "get # of tasks" vs "iterate over them"

wfchandler force-pushed the wc/bundle-sp-dumps branch from 7380a0e to a4b0acf Compare May 16, 2025 14:49

Capture SP task dumps in support bundles

dbb6669

Update the support bundle collector to capture task dumps from the SPs.

wfchandler force-pushed the wc/bundle-sp-dumps branch from a4b0acf to dbb6669 Compare May 18, 2025 23:05

wfchandler commented May 20, 2025

View reviewed changes

wfchandler marked this pull request as ready for review May 20, 2025 20:55

wfchandler requested review from papertigers and smklein and removed request for papertigers May 20, 2025 20:55

smklein reviewed May 20, 2025

View reviewed changes

wfchandler added 4 commits May 21, 2025 10:16

Merge branch 'main' into wc/bundle-sp-dumps

50457f7

Switch to ParallelTaskSet and show SPs in report

fc341e1

Assert sp dump dir is created

132e7bb

Fix name assertion

6b2892d

wfchandler requested a review from smklein May 21, 2025 23:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture SP task dumps in support bundles #8177

Capture SP task dumps in support bundles #8177

wfchandler commented May 16, 2025

wfchandler commented May 20, 2025

wfchandler May 20, 2025 •

edited

Loading

smklein May 20, 2025

smklein May 20, 2025

smklein May 20, 2025

smklein May 20, 2025

smklein May 20, 2025

smklein May 20, 2025

Capture SP task dumps in support bundles #8177

Are you sure you want to change the base?

Capture SP task dumps in support bundles #8177

Conversation

wfchandler commented May 16, 2025

wfchandler commented May 20, 2025

wfchandler May 20, 2025 • edited Loading

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

smklein May 20, 2025

Choose a reason for hiding this comment

wfchandler May 20, 2025 •

edited

Loading