Fix linux cpu builds. Resolves nightly build for mac stops on 0422

[atalman]

Fixes #2157

This reverts 2 PRS:
#2044
#2093
Which introduced the regression.

This works: torchao-0.11.0.dev20250505+cpu-py3-none-any.whl generates correct file.
Please see details on #2157

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2170

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• CI workflows being skipped on PR

❌ 1 New Failure

As of commit 53b4b28 with merge base 94e2e05 ():

NEW FAILURE - The following job has failed:

• Run TorchAO Experimental Tests / test-mps-ops (macos-m1-stable) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.