Automatically add aliases for expect tests

[johnridesabike]

Desired Behavior

For Dune's built-in expect tests, where a test executable prints to stdout and Dune diffs the output with a .expected file, it would be useful if Dune provided an alias for each test.

Expect test diffs can be large so the ability to specify a single test would be useful. Right now, the options to execute a single expect test are either to put them in separate directories or manually define each with a rule instead of the (tests ...) stanza.

Dune already provides aliases for each cram test, so extending that feature to expect tests feels natural.

Example

Directory:

.
└── test
    ├── dune
    ├── test_1.expected
    ├── test_1.ml
    ├── test_2.expected
    └── test_2.ml

Contents of test/dune:

(tests (names test_1 test_2))

Commands:

$ dune build @test_1 # executes and diffs test_1
$ dune build @test_2 # executes and diffs test_2

[emillon]

Adding automatic aliases seems a bit too much, but my suggestion from elsewhere was to add (alias) and (runtest_alias) support to the test stanza.

[rgrinberg]

I don't see the issue with adding more aliases (what's the downside?), but in your example you can already execute your tests individually by building the expected target. E.g. dune build test_1.expected.

[johnridesabike]

in your example you can already execute your tests individually by building the expected target. E.g. dune build test_1.expected.

This does not seem to be the same thing. Running dune build example.expected just copies example.expected to the build directory. It doesn't generate the accompanying .output file and diff the two.

I like the idea of adding (alias) to the test stanza too. My main reasoning for creating the aliases automatically is because that's how cram tests work already, so it seems like it would be consistent with that.

[rgrinberg]

Okay, then the command should be dune build example.output or whatever. You just need to build the target that is produced by your command and doesn't live in the source tree.

[johnridesabike]

Building an .output target will produce the file but print nothing and exit with status 0 even if the test should fail. It also does not allow promoting the output to the source tree's .expected file. The actual test comes from diffing the output with the expected file, which only exists in the runtest alias.

Here's a minimal example of what I'm describing.

Files:

dune-project:
(lang dune 3.14)

dune:
(test (name example))

example.ml:
let () = print_endline "fail"

example.expected:
pass

Commands:

$ dune build example.expected
$ dune build example.output
$ dune build @runtest
File "example.expected", line 1, characters 0-0:
------ example.expected
++++++ example.output
File "example.expected", line 1, characters 0-1:
-|pass
+|fail

The output of dune build @runtest is what we want, but with the added ability to specify an individual test (if there were multiple tests).

[rgrinberg]

Okay, that's convincing. Feel free to send a PR to add an alias