metriq-gym

metriq-gym is a Python framework for implementing and running standard quantum benchmarks on different quantum devices by different providers.

• Open – Open-source since its inception and fully developed in public.
• Transparent – All benchmark parameters are defined in a schema file and the benchmark code is reviewable by the community.
• Cross-platform – Supports running benchmarks on multiple quantum hardware providers (integration powered by qBraid-SDK)
• User-friendly – Provides a simple command-line interface for dispatching, monitoring, and polling benchmark jobs (you can go on with your life while your job waits in the queue).

Data generated by metriq-gym is intended to be published on https://metriq.info.

Join the conversation!

• For code, repo, or theory questions, especially those requiring more detailed responses, submit a Discussion.
• For casual or time sensitive questions, chat with us on Discord's #metriq channel.

Setup

You will require Python 3.12 or above, and poetry.

Cloning the repo

When cloning the metriq-gym repository use:

git clone --recurse-submodules https://github.com/unitaryfund/metriq-gym.git

This allows you to fetch qiskit-device-benchmarking as a git submodule for a set of some of the IBM benchmarks.

Installation

Once you have poetry installed and the repository cloned, run:

poetry install

from the root folder of the project, in order to install the project dependencies. We recommend doing this in an isolated virtual environment. See Poetry documentation for more information on managing virtual environments.

If you use pyenv, here is a quick start guide to set up the environment and install all dependencies:

pyenv install 3.13  
pyenv local 3.13 
poetry install
eval $(poetry env activate)

All Python commands below should be run in the virtual environment.

Running benchmarks

Credential management

To run on hardware, each hardware provider offers API tokens that are required to interact with their quantum devices. In order to run on these devices, you will need to follow the instructions on the respective pages of the providers and obtain API keys from them.

The .env.example file illustrates how to specify the API keys once you have acquired them. You will need to create a .env file in the same directory as .env.example and populate the values of these variables accordingly.

Workflow

You can dispatch a job by specifying the parameters of the job you wish to launch in a configuration file.

python metriq_gym/run.py dispatch <BENCHMARK_JSON> --provider <PROVIDER> --device <DEVICE>

Refer to the schemas/ directory for example schema files for other supported benchmarks.

If running on quantum cloud hardware, the job will be added to a polling queue. The status of the queue can be checked with

python metriq_gym/run.py poll --job_id <METRIQ_GYM_JOB_ID>

where <METRIQ_GYM_JOB_ID> is the assigned job ID of the job that was dispatched as provided by metriq-gym.

Alternatively, the poll action can be used without the --job_id flag to view all dispatched jobs, and select the one that is of interest.

python metriq_gym/run.py poll

View jobs

You can view all the jobs that have been dispatched by using the view action. This will display basic information about each job, including its ID, backend, job type, provider, and device.

python metriq_gym/run.py view

In order to view the details of a specific job (e.g., the parameters the job was launched with), you can use the view action with the --job_id flag or select the job by index from the list of all dispatched jobs.

python metriq_gym/run.py view --job_id <METRIQ_GYM_JOB_ID>

Example: Benchmarking Bell state effective qubits (BSEQ) on IBM hardware

The following example is for IBM, but the general workflow is applicable to any of the supported providers and benchmarks.

To run on IBM hardware, you will also require an IBM token. To obtain this, please visit the IBM Quantum Platform and include the API token in the local .env file as previously described.

The schemas/examples/ directory houses example JSON configuration files that define the benchmark to run. In this case, we use the bseq_example.json file as we want to run a BSEQ job. The following dispatches a job on the ibm-sherbrooke device for BSEQ.

python metriq_gym/run.py dispatch metriq_gym/schemas/examples/bseq.example.json --provider ibm --device ibm_sherbrooke

We should see logging information in our terminal to indicate that the dispatch action is taking place:

INFO - Starting job dispatch...
INFO - Dispatching BSEQ benchmark job on ibm_sherbrooke device...
...
INFO - Job dispatched with ID: 93a06a18-41d8-475a-a030-339fbf3accb9

We can confirm that the job has indeed been dispatched and retrieve the associated metriq-gym job ID (along with other pieces of metadata).

+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+
| Metriq-gym Job Id                    | Provider   | Device                                               | Type           | Dispatch time (UTC)        |
+======================================+============+======================================================+================+============================+
| 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm        | ibm_sherbrooke                                       | BSEQ           | 2025-03-05T10:21:18.333703 |
+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+

We can use the "poll" action to check the status of our job:

python metriq_gym/run.py poll --job_id 93a06a18-41d8-475a-a030-339fbf3accb9

Doing so gives us the results of our job (if it has completed):

INFO - Polling job...
BSEQResult(largest_connected_size=100, fraction_connected=0.7874015748031497)

In the event where the job has not completed, we would receive the following message instead

INFO - Polling job...
INFO - Job is not yet completed. Please try again later.

As a convenience, while we could supply the metriq-gym job ID, we can also poll the job by running python metriq_gym/run.py poll and then selecting the job to poll by index from our local metriq-gym jobs database.

Available jobs:
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
|    | Metriq-gym Job Id                    | Provider   | Device                                               | Type           | Dispatch time (UTC)         |
+====+======================================+============+======================================================+================+=============================+
| 0  | 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm        | ibm_sherbrooke                                        | BSEQ           | 2025-03-05T10:21:18.333703 |
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
Select a job index: 

Entering the index (in this case, 0), polls the same job.

Select a job index: 0
INFO - Polling job...

Contributing

First, follow the Setup instructions above.

Updating the submodule

To pull the latest changes from the submodule’s repository:

cd submodules/qiskit-device-benchmarking
git pull origin main

Then, commit the updated submodule reference in your main repository.

Style guide

We don't have a style guide per se, but we recommend that both linter and formatter are run before each commit. In order to guarantee that, please install the pre-commit hook with

poetry run pre-commit install

immediately upon cloning the repository.

Tests

The suite of unit tests can be run with

poetry run pytest

Type checking

The project uses mypy for static type checking. To run mypy, use the following command:

poetry run mypy

Documentation

The project uses Sphinx to generate documentation. To build the HTML documentation:

1.Navigate to the docs/ directory:

cd docs/

Run the following command to build the HTML files:

make html

Open the generated index.html file located in the _build/html/ directory to view the documentation.

Data

Benchmark schema is defined here.

First hardware results are here.