use bashunit assertions outside of test scripts

[staabm]

we are in the process of integrating bashunit into the PHPStan project.

see phpstan/phpstan-src#3160 for the initial draft in which I try to convert a few existing tests into a bashunit

---

our primary goal is to make our existing tests more readable (for people not used to bash).

e.g. one of our current tests look like:

cd e2e/trait-caching
../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/
patch -b data/TraitOne.php < TraitOne.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 1 ]
grep 'Method TraitsCachingIssue\\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"

two things we want to improve on this test:

1. grep is not readable.. atm one needs to be aware of chars to escape.. assert_contains is way better

could we use the bashunit assertions outside a test? e.g. bashunit assert_contains "my needle" "$OUTPUT" ?
is such construction somehow supported? or maybe we can "import" bashunit assertion functions into the current bash, so we can just call them with assert_contains?

2. using a more readable variant of counting lines in $OUTPUT. is it possible to register custom assertion functions? or would a new core assertion function like assert_line_count be possible?

[ondrejmirtes]

Basically we want to use the nice assertions syntax of bashunit not to unit tests bash functions, but to end-to-end tests executables.

Is this supported by bashunit? Do you have any tips for us?

I've been following the project for a long time and I'm a huge fan of your approach (release early and often)!

[Chemaclass]

@staabm @ondrejmirtes I like the idea of implementing a feature to allow using bashunit functions isolated without the tests. Actually, you already prepared a very nice starting point here https://github.com/phpstan/phpstan-src/pull/3160/files#r1641646749
I will work out something today focus on the DX integrated on the bashunit core itself

[ondrejmirtes]

That would be awesome! Also it will make you cramp more content into your bashunit talk next time you give it I bet 😀

[Chemaclass]

About the second point of the issue, I think it would be easily doable adding that new assert function to count the total lines of a given string. Wanna try yourself a PR? We will help you if you need any support. You can use other assert functions as examples :)

[staabm]

I will look into it

[Chemaclass]

@staabm @ondrejmirtes I am trying to think a good name for this feature in the website/docs, but I am not sure how to call it. What would you suggest? I could start with some draft, and you could also feel free to improve it adding more context (using your use-case as real example that could help people realise the potential of this feature). How does this sounds to you?

[ondrejmirtes]

If I were to describe it, it's readable and comfortable assertions for integration or e2e tests, for entire applications or executables.

You can show that alongside Bash examples of achieving the same thing without bashunit (both test code and output), and see how much less readable and understandable that is.

[ondrejmirtes]

More ideas:

• External assertions
• Standalone assertions

[Chemaclass]

Alright, I used the "Standalone" term for the initial idea/template (thanks, @ondrejmirtes). Let see what you all think on the text and if someone come up with a better naming we can change it, in this PR -with a review-, or in a follow up one. The PR is not a draft anymore, but something potential to be merged if the team and contributors agree! 🚀

[Chemaclass]

@ondrejmirtes @staabm this new feature is already implemented (#258) and I like it very much. However, I was wondering why you didn't want to use the normal bashunit runner, wrapping your e2e tests into test functions? Is there a specific reason for that or what was/is preventing you to do that?

[ondrejmirtes]

Right now we have this structure: https://github.com/phpstan/phpstan-src/blob/108277a6348de38eaaeb6a5bdb89f83dd775fc9c/.github/workflows/e2e-tests.yml#L191-L216

Each of these items gets separate and clean environment, they are executed as a separate job. I don't see a way how to rewrite this into bashunit functions and keep these advantages.

[Chemaclass]

Done & Merged

• Allow calling assertions outside tests Allow calling assertions outside tests #258
• Add assert_line_count Add assert_line_count #259