Normalize File Paths In Generated Reports (Linux / Windows)

[abbotware]

I run unit tests on windows and linux. This produces coverage report files that have different path roots:

C:\src\build\file.cs
vs
\var\src\build\file.cs

When the reports are merged it thinks there are 2 sets of functions. Instead of 100% code coverage, i have 50% coverage.

Perhaps a new command line option is needed to normalize the path:

coverlet -stripFromPath=C:\src\
coverlet -stripFromPath=\var\

That way both reports will then have a normalized file

[MarcoRossignoli]

Which format are you using?Which reporter?
Let me understand your scenario, why you have to merge two coverage from two different plat?
I mean if you've some code that run on win and other that run on unix system you could filter out files with ExcludeByFile filter.

[abbotware]

Cobetrua format. The msbuild integration tasks.

I write software that has to work on different platforms. There are subtle differences in behavior on different OS platforms and I have caught some bugs by just running the same code on different platforms. Sometimes this is known ahead of time, other times it is not - so I can't filter by linux /
windows.

Even if I did it would be the same function with different code branches... and hence I would want to see the merged result of coverage between linux and windows

[abbotware]

Also from reading more of the Docs.. it looks like if I was using SourceLink (I am not using git so I can't) it might resolve the paths to the same file and this would not be an issue.

so in my case, I think there are 2 options:

1. make the report merger smarter (ie use namespace / class / function to align code)
2. normalize the file names in the reports (I think this is the easier option)

however if you know any other way please let me know!

Also I am more than happy to make an attempt via a branch / PR for option 2 as it looks like it would just be some new option that is used to normalize the files before any processing really begins.

[MarcoRossignoli]

With #614 in next version you could only update "sources" is you've same relative path for cs.

Also I am more than happy to make an attempt via a branch / PR for option 2 as it looks like it would just be some new option that is used to normalize the files before any processing really begins.

I don't know...seem very specific to your use case, I mean you've "same" relative path until root, there are tons of other scenario where this rule don't work.
For now if you know well your report you could simply create a script that renames fragment how you want before generate reports.

[abbotware]

so then your saying you don't support coverage reports on linux and windows at the same time - The paths will never be the same since they use a different convention.

[abbotware]

But you are right about the pre-processing script... I can do an xslt or regex replacement easy enough :-)

[abbotware]

so upon doing this, the counts seem wrong - each code block is only hit 1 time even though I know each unit test was run twice! (linux and windows)

Other problems:

• the mainreport says 100% code / 50% branch - but the drill down says 100% for both
• the branch covered icon has the green / red indicator meaning that a branch was not covered

Editing the paths in the report is effectively the same thing as if I were to add the option I was suggesting, so I think these issues are more internal.

[MarcoRossignoli]

so then your saying you don't support coverage reports on linux and windows at the same time

At same time no...it's first time I see a this request, but it's interesting, usually this is stuff for "post report production" related to report viewers.

Other problems:

Can you provide a repro?Or provide the "json" version of report?Json version is the "raw" version that we can use to troubleshoot.

[abbotware]

oh wait a minute - while yes I originally requested the cmd line option in coverlet to produce normalized files... the report is being merged by dotnet-reportgenerator-cli ... so those are not problems for you :-)

[MarcoRossignoli]

@abbotware feel free to close this issue if solved from our side.

[abbotware]

@MarcoRossignoli - it appears that the output files have different branch data for the same lines of code - so when merged the counts are incorrect.

@danielpalme discovered the inconsistency when I sent him the raw input files for analysis

in one case 1/1 branches are covered... in another case 0/2 branches are covered - for the same exact line of code!

[abbotware]

I have a unit test and integration test stage - the unit tests are 'safe' to run at all times.. the integration tests may have side effects and interact with 3rd party systems...

It would make sense that in one coverage report (the unit tests) there would be some code that is never executed (the 0/2 branch) while the integration test will execute that code (the 1/1 branch)

why the branch counts are different seems to be the issue

[MarcoRossignoli]

It would make sense that in one coverage report (the unit tests) there would be some code that is never executed (the 0/2 branch) while the integration test will execute that code (the 1/1 branch)

Are you sure to compile/run with same config (debug/release)?The only thing comes to mind is that different build config could generate different IL an different instrumentation, but to understand we should isolate that piece of code and run instrumentation/test with different config, compiler could emit different IL. Again if for instance in one test you await same code in different place it's possible that the generated state machine is very different despite same similar c# code.

[abbotware]

both are release builds.. this is async/await related code.

The one thing I can think is that i am doing a dotnet test and in both cases it is performing a restore / build - so the binaries being tested are technically different - but for the same exact version of source code and same build settings. (will double check though)

I could possibly make a minimal example and see if its easily reproducible.

The other thing I can think of is JIT related. Do you know if the jitter does anything different if the assembly is not in any code paths?

[MarcoRossignoli]

The other thing I can think of is JIT related. Do you know if the jitter does anything different if the assembly is not in any code paths?

AFAIK jitter imports and jits only code that is "used", and btw also with inlining or other optimization the "hit record" path is simply a +1 inside an array...so I don't expect missing thing due to optimization, code could be optimized but not skipped.

[abbotware]

@MarcoRossignoli - I had a thought - this might be a compiler issue - in one case the code is being built via the linux SDK and the other windows SDK.

I am setting the SDK version via global.json so it does use the same version of the SDK, but perhaps there really is some difference in the generated IL.

I will grab the assemblies and inspect the generated IL.

The libraries are net standard, and I do precompile nugets, but I don't use the nugets for unit tests. Please correct me if I am wrong, but I thought the best way to get valid code coverage would be to do a dotnet test where the unit tests builds all the code. The build system ensures the source code version matches the the upstream build stage so I know the source is in sync.

[abbotware]

I never liked doing a dotnet test where the I was building the assemblies (since I know I am not actually 'testing' the binaries being shipped)

It might be more work on my part - but If the unit tests consume the nugets (instead of building) -

1. would I still get valid code coverage?
2. If I ensure one OS builds the nugets, will this normalize the paths ( the reason I opened this issue! :-) )

Doing this would obviously eliminate any IL differences

[MarcoRossignoli]

I am setting the SDK version via global.json so it does use the same version of the SDK, but perhaps there really is some difference in the generated IL.

I would say not...btw only IL inspection can answer. I mean roslyn is a cross compiler so I expect same output IL.

The libraries are net standard, and I do precompile nugets, but I don't use the nugets for unit tests. Please correct me if I am wrong, but I thought the best way to get valid code coverage would be to do a dotnet test where the unit tests builds all the code.

Yes this is the correct way to do...to speedup ci usually you can do separate steps like

dotnet restore
dotnet build --no-restore
dotnet test --no-build

It might be more work on my part - but If the unit tests consume the nugets (instead of building) -
would I still get valid code coverage?
If I ensure one OS builds the nugets, will this normalize the paths ( the reason I opened this issue! :-) )

You shouldn't do this for coverage, you have to run coverage on code that will end inside a nuget package, because coverlet can instrument only dll where local pdb/cs files are present, I don't think it's the case with "packaged" assets.

If I ensure one OS builds the nugets, will this normalize the paths ( the reason I opened this issue! :-) )

If you can ensure that tests cover all code on one OS you'll have only one coverage file, but if your test are different for different OS it's not possible and you have to "manual merge" unix/win files at the end and put this inside correct folder place where report generator can find sources.

both are release builds.. this is async/await related code.

Could be great try to repro the difference to understand if it's a "coverage issue"(we have a lot of improvement in next release on async/await coverage https://github.com/tonerdo/coverlet/blob/master/Documentation/Changelog.md#unreleased ) or a difference in tests run on different OS.

[abbotware]

The IL looks the same - I compared one of the functions with different branch counts across 4 variations: Windows vs Linux and unit test vs Integration test

right now with manually updating the paths - the report files merge together nicely -

the only issue is the branch count is different on the async / await code so perhaps the next release will address this?

is there an pre-release nuget I can test out?

[MarcoRossignoli]

is there an pre-release nuget I can test out?

Yes we have nightly build https://github.com/tonerdo/coverlet/blob/master/Documentation/ConsumeNightlyBuild.md

The other way is to try to repro with similar code.

[abbotware]

The other way is to try to repro with similar code.

trying to isolate the code see snippet below:

OK some good news? I am looking at one of the functions that had a problem and now at least the total branch counts match (inspecting the output xmls)!

I am still only getting 50% coverage - but perhaps it really is only 50%? take the following code for example:

        public async Task<ActionResult<IEnumerable<SystemUserItem>>> SystemUsers(CancellationToken ct)
        {
            using var data = this.factory.CreateReadOnly<SystemUserItem>();

            var list = await data.AllAsync(ct)
                .ConfigureAwait(false);

            return this.Ok(list);
        }

FYI - with the nightly build - using the msbuild version

1. it seems the output file path patterns have changed - I am using cobetura and it seems that cobutera.xml was appended to the output of the files... which broke my search patterns.
2. is there a way to print the version # of coverlet so I know what versionis running ? (ie like a banner)

[abbotware]

Also - looks like #661 actually does the path normalization? :-)

[abbotware]

for the above code, here is the associated IL Code

il code.txt

[MarcoRossignoli]

it seems the output file path patterns have changed - I am using cobetura and it seems that cobutera.xml was appended to the output of the files... which broke my search pattern

We did some change on file generation to allow multi-target name generation #636 ...how do you run code?Can you attach your commandline?

[MarcoRossignoli]

I am still only getting 50% coverage - but perhaps it really is only 50%? take the following code for example:

Where do you see 50%?On ConfigureAwait line?

[abbotware]

here is the final report output:

here is the diff of the coverage xml:

[abbotware]

how do you run code?Can you attach your commandline?

dotnet test $Source --filter $TestFilter -f $Framework -c $Configuration /p:NetCoreSdk=$NetCoreSdk --logger:"trx;LogFileName=$TrxFile" --results-directory:$TestResultsPath /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$TestResultsPath$CoverageDataFile /p:CollectCoverage=true /p:ExcludeByAttribute="Obsolete%2cGeneratedCodeAttribute%2cCompilerGeneratedAttribute" -nodeReuse:false

where $CoverageDataFile=project.Release.linux-x64.netcoreapp3.1.coverage.xml
and for some reason the output file is actually:
project.Release.linux-x64.netcoreapp3.1coverage.netcoreapp3.1.xml.cobertura.xml

this is a bug since I am specifying a single framework via -f
#667

[MarcoRossignoli]

where $CoverageDataFile=project.Release.linux-x64.netcoreapp3.1.coverage.xml
and for some reason the output file is actually:
project.Release.linux-x64.netcoreapp3.1coverage.netcoreapp3.1.xml.cobertura.xml

@abbotware for file naming issue I fixed to be back-compat, please tomorrow can you try nightly?

Now we keep you file name if specified file name has got extension.

this is a bug since I am specifying a single framework via -f

Unfortunately this is not a bug, if you're using TargetFramewoks we cannot know if you're passing -f and we have only the knowledge that this is a multitarget project and we're "building" that specific version.
You can check this new tests

https://github.com/tonerdo/coverlet/blob/master/test/coverlet.integration.tests/Msbuild.cs#L62
https://github.com/tonerdo/coverlet/blob/master/test/coverlet.integration.tests/Msbuild.cs#L140

So you'll get project.Release.linux-x64.netcoreapp3.1coverage.netcoreapp3.1.xml

[MarcoRossignoli]

here is the diff of the coverage xml:

From this report seem that for left one no test are running on that part of code, for right one seem an issue with using+async await+ConfigureAwait, I'll try to understand if new using syntax changed a bit code generated(maybe some new null check are emitted)

If you can isolate a simple sample(out of your classes) that show this behaviour would be very helpful.

[MarcoRossignoli]

@abbotware please can you open a new issue to track #649 (comment)?
This one is related to report file normalization that's not an issue, I mean different OS will generate different path format and it's ok.