[DOCS-10234] Test Health (new page) #28926
Open
+132
−21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Preview links (active after the
|
|
Created DOCS-10793 for Docs Team review. |
jhgilbert
approved these changes
May 9, 2025
| Based on the current time frame and filters applied, the dashboard highlights the following key metrics: | ||
|
|
||
| - [**Pipelines Failed**](#pipelines-failed): Sum total of pipelines that failed due to flaky tests | ||
| - [**Time Wasted in CI**](#time-wasted-in-ci): Total time wasted in CI due to flaky tests |
There was a problem hiding this comment.
Suggested change
| - [**Time Wasted in CI**](#time-wasted-in-ci): Total time wasted in CI due to flaky tests | |
| - [**Time Wasted in CI**](#time-wasted-in-ci): Total time spent in CI due to flaky tests |
Just to avoid echoing the term in the definition
| |--------|-------------| | ||
| | **Pipeline Executions with Tests** | Number of pipeline executions with one or more test sessions. | | ||
| | **Failures Due to Flaky Tests** | Number of pipeline executions that failed solely due to flaky tests. All tests that failed have one or more of the following tags: `@test.is_known_flaky` or `@test.is_new_flaky`. | | ||
| | **Failures Due to Non-Flaky Tests** | Number of pipeline executions that failed due to tests without any flakiness. All tests that failed do not have any of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, and `@test.is_flaky`. | |
There was a problem hiding this comment.
Suggested change
| | **Failures Due to Non-Flaky Tests** | Number of pipeline executions that failed due to tests without any flakiness. All tests that failed do not have any of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, and `@test.is_flaky`. | | |
| | **Failures Due to Non-Flaky Tests** | Number of pipeline executions that failed due to tests without any flakiness. None of the failing tests have any of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, and `@test.is_flaky`. | |
| ### Enhance developer experience | ||
| Use **Dev Experience - Test Failure Breakdown** and **Dev Experience - Time Lost Breakdown** to identify how often flaky tests in particular cause failures and waste CI time. | ||
|
|
||
| These Test Optimization features improve developer experience by alleviating test failures and wasted time: |
There was a problem hiding this comment.
Suggested change
| These Test Optimization features improve developer experience by alleviating test failures and wasted time: | |
| These Test Optimization features improve developer experience by reducing test failures and wasted time: |
Just to use a less fancy word :)
| - **[Test Impact Analysis][4]** minimizes the flaky tests that run, by only running relevant tests based on code coverage. Skipping irrelevant tests also shortens the feedback loop for developers. | ||
|
|
||
| ### Maximize pipeline efficiency and reduce costs | ||
| Lengthy test suites slow down feedback loops to developers and running irrelevant tests incurs unnecessary costs. |
There was a problem hiding this comment.
Suggested change
| Lengthy test suites slow down feedback loops to developers and running irrelevant tests incurs unnecessary costs. | |
| Lengthy test suites slow down feedback loops to developers, and running irrelevant tests incurs unnecessary costs. |
There was a problem hiding this comment.
It's not a big enough deal to fix, but this screenshot is on the small side in terms of readability. Next time, you might tap "Ctrl +" a time or two in order to cut down on the amount of white space in the shot, making the text bigger and easier to read without the image needing to be any larger.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Test Health is a net-new tab within Test Optimization. It helps users (typically platform engineers or devops) track and improve their CI tests.
Merge instructions
Note
This PR intentionally ignores some Vale warnings: several headings use title case because they're matching labeled UI elements, and in one spot future tense actually seems appropriate because it's about a "yet to be identified" use case.
Merge readiness:
For Datadog employees:
Merge queue is enabled in this repo. Your branch name MUST follow the
<name>/<description>convention and include the forward slash (/). Without this format, your pull request will not pass in CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.If your branch doesn't follow this format, rename it or create a new branch and PR.
To have your PR automatically merged after it receives the required reviews, add the following PR comment:
Additional notes