feat: add soft assertions feature

[JustasMonkev]

This pull request introduces a new "soft assertions" feature to the WebdriverIO testing framework, allowing tests to continue execution even when some assertions fail. It includes updates to the documentation, core library, and new utility classes for managing and integrating soft assertions.

New Feature: Soft Assertions

Documentation Updates:

• Added a new section in docs/API.md explaining the concept of soft assertions, usage examples, and API methods (expect.soft, expect.getSoftFailures, expect.assertSoftFailures, expect.clearSoftFailures). Also included integration details with WebdriverIO's test runner.

Core Library Enhancements:

• Updated src/index.ts to extend the expect object with soft assertion methods (soft, getSoftFailures, assertSoftFailures, clearSoftFailures) and exported the new soft assertion services. [1] [2] [3]

Soft Assertion Utilities:

• Introduced SoftAssertService in src/softAssert.ts to manage soft assertion failures per test, including methods to add, retrieve, clear, and assert failures. It uses a singleton pattern for centralized failure tracking.
• Added SoftAssertionService in src/softAssertService.ts for WebdriverIO integration, hooking into test and step lifecycle events to automatically handle soft assertion failures.

Soft Assertion API:

• Created createSoftExpect in src/softExpect.ts, which generates soft assertion matchers using proxies. It ensures that failures are recorded without stopping test execution.

Closes #227

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: JustasMonkev / name: JustasM (27af273, 11aced2, 13cf6ca, 8dbf1ff)
• ✅ login: christian-bromann / name: Christian Bromann (2547329, cecd54a, 06c5d39)

[christian-bromann]

Great contribution 👏 some comments here

[christian-bromann]

@darkartswizard @PipulPant @DivyangRaval @circa1741 and @alfonso-presa curious about your feedback here 🤔

[darkartswizard]

Soft assertions is my No. 1 goto question regarding test automation frameworks. The main advantage is that I can perform multiple validations on a page, such as a shipping address, Pass or Fail, without having to re-run the test multiple times, if the first field fails, to catch the next error. The only time I would do a Hard assertion is if the element I'm trying to click does not exist such as "Add to Cart" button. One suggestion would be an option for an implicit *assertAll()* so that it does not have to be in every test. I know it would normally be added to an *afterTest()* hook. But it would be nice if an softAssert() was called anywhere, the assertAll() would be implicitly called to collect the results. The other major challenge I see might be ensuring the step results are reported, with screen captures, in correct numerical order. Lastly, when I design my frameworks, SoftAsserts come in handy as I create wrappers for all commands. When a Hard Assert occurs all the remaining steps execute but will null out, recoding only the name of the step that would have been executed. This gives a consistent total count of steps / progress in the test results, regardless of when the test failed. Best regards, Paul

…

On Sun, May 25, 2025 at 2:04 PM Christian Bromann ***@***.***> wrote: *christian-bromann* left a comment (webdriverio/expect-webdriverio#1836) <#1836 (comment)> @darkartswizard <https://github.com/darkartswizard> @PipulPant <https://github.com/PipulPant> @DivyangRaval <https://github.com/DivyangRaval> @circa1741 <https://github.com/circa1741> and @alfonso-presa <https://github.com/alfonso-presa> curious about your feedback here 🤔 — Reply to this email directly, view it on GitHub <#1836 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMDP3YNWCBUIJIVIFTACVOD3AIH2TAVCNFSM6AAAAAB5ZIDFPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMBYGAYDSNJXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[alfonso-presa]

Hi! From a test developer's perspective, I'm not entirely sure that best the way to implement soft assertions is to generate a new syntax for expect... Assertions are sometimes (a lot of the times to be honest) implicit, e.g. if the element I'm looking for isn't on the page... I'm more inclined to wrap the section of code where you want assertion failures to be catched and kept for later so that the test continues to the end and then failure is reported as a soft assertion indicating the step were it happened.

In this example

await expect.soft(await $('h1').getText()).toEqual('Basketball Shoes')

If no h1 is on the page, I guess the failure would make the test stop. I'd rather have something like:

await expect.soft(() => $('h1').getText()).toEqual('Basketball Shoes')

Or IMHO, more flexible, but more verbose:

await soft(async () => {
   expect(await $('h1').getText()).toEqual('Basketball Shoes'))
});

I know for an h1 is not clear the use case, but there are situations where the page structure changes depending on the circumstances (for example, in a bank application if a expense has no category), so I think soft assertions should cover also the implicit use cases.

Cheers!

[JustasMonkev]

Hi! From a test developer's perspective, I'm not entirely sure that best the way to implement soft assertions is to generate a new syntax for expect... Assertions are sometimes (a lot of the times to be honest) implicit, e.g. if the element I'm looking for isn't on the page... I'm more inclined to wrap the section of code where you want assertion failures to be catched and kept for later so that the test continues to the end and then failure is reported as a soft assertion indicating the step were it happened.

In this example

await expect.soft(await $('h1').getText()).toEqual('Basketball Shoes')

If no h1 is on the page, I guess the failure would make the test stop. I'd rather have something like:

await expect.soft(() => $('h1').getText()).toEqual('Basketball Shoes')

Or IMHO, more flexible, but more verbose:

await soft(async () => {
   expect(await $('h1').getText()).toEqual('Basketball Shoes'))
});

I know for an h1 is not clear the use case, but there are situations where the page structure changes depending on the circumstances (for example, in a bank application if a expense has no category), so I think soft assertions should cover also the implicit use cases.

Cheers!

Hey! You make a good point about implicit failures.

My expect.soft() approach follows Playwright's pattern, which developers already know. Your wrapper idea would definitely catch more edge cases like when elements don't exist.

Let's try this - if people suggest use cases like yours, we can update the implementation to handle them. For now, I'd suggest keeping it as currently implemented with expect.soft() since it covers the main scenarios and matches what developers expect from other frameworks.

[christian-bromann]

Awesome 👍

[christian-bromann]

Fixing the build

[christian-bromann]

@JustasMonkev mind taking a look at the build? Once test pass I am happy to merge this!

[JustasMonkev]

@JustasMonkev mind taking a look at the build? Once test pass I am happy to merge this!

Yes, on it checking

[JustasMonkev]

@christian-bromann I checked and it looks like soft assertion tests are passing, but the failures are the same as in main. Should I fix it here or open a new PR?

[christian-bromann]

Should I fix it here or open a new PR?

Oh, yeah that would be great!

[JustasMonkev]

[JustasMonkev]

@christian-bromann could you rerun the tests here?

[JustasMonkev]

@christian-bromann tests passed but 😄

https://github.com/webdriverio/expect-webdriverio/actions/runs/15279626306/job/42975532806?pr=1836#step:6:257

[christian-bromann]

@JustasMonkev oh damn, let's adjust the coverage treshold, I think this PR has sufficient tests.

[JustasMonkev]

@JustasMonkev oh damn, let's adjust the coverage treshold, I think this PR has sufficient tests.

@christian-bromann done

[christian-bromann]

Awesome, LGTM 👍

[wdio-bot]

Hey JustasMonkev 👋

Thank you for your contribution to WebdriverIO! Your pull request has been marked as an "Expensable" contribution.

We've sent you an email with further instructions on how to claim your expenses from our development fund.
Please make sure to check your spam folder as well. If you have any questions, feel free to reach out to us at expense@webdriver.io or in the contributing channel on Discord.

We are looking forward to more contributions from you in the future 🙌

Have a nice day,
The WebdriverIO Team 🤖