Skip to content

Running coverlet.msbuild inside docker with --no-build #1121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pankti11 opened this issue Mar 9, 2021 · 8 comments
Closed

Running coverlet.msbuild inside docker with --no-build #1121

pankti11 opened this issue Mar 9, 2021 · 8 comments
Labels
stale waiting for customer Waiting for customer action

Comments

@pankti11
Copy link

pankti11 commented Mar 9, 2021 •
edited
Loading

I am trying to run coverlet.msbuild with --no-build flag inside the docker container. And I am never able to generate the code coverage. I cannot run restore in the container. Hence, container does not have an access coverlet.msbuild nuget.

Logs:
Command:
dotnet test $startDir/$testPath/$testFile /p:CollectCoverage=true /p:CoverletOutputFormat=json /p:CoverletOutput=$outputFile /p:MergeWith=$mergeFile --verbosity diag --no-build

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed LogDataConverterMapsLogLevelsToLogSeverities [15 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (None,Unknown) [9 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Trace,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Debug,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Information,Information) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Warning,Warning) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Error,Error) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Critical,Error) [< 1 ms]
  Passed LogDataConverterDetectsUnknownLogLevelMappings [< 1 ms]
  Passed LogStoreProviderEnumDefinedTest [148 ms]
  Passed LogStoreProviderEnumNotDefinedTest [< 1 ms]

Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.1038 Seconds
                   Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask". (TaskId:21)
21:17:54.857     1>Done building target "VSTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:2)
21:17:54.858     1>Done Building Project "/github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (VSTest target(s)).
Deferred Messages
21:17:54.861     0>
Detailed Build Summary
======================
     (TaskId:0)
                     
                     ============================== Build Hierarchy (IDs represent configurations) =====================================================
                     Id                  : Exclusive Time   Total Time   Path (Targets)
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     0                   : 1.768s           1.768s       /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj (VSTest)  (TaskId:0)
                     
                     ============================== Node Utilization (IDs represent configurations) ====================================================
                     Timestamp:            1        Duration   Cumulative
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     637509214730885310:   0        1.772s     1.772s ................................... (TaskId:0)
                     -----------------------------------------------------------------------------------------------------------------------------------
                     Utilization:          100.0    Average Utilization: 100.0 (TaskId:0)

Project Evaluation Performance Summary:
      290 ms  /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls

Project Performance Summary:
     1428 ms  /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls
               1428 ms  VSTest                                     1 calls

Target Performance Summary:
        9 ms  ShowCallOfVSTestTaskWithParameter          1 calls
     1423 ms  VSTest                                     1 calls

Task Performance Summary:
        8 ms  Message                                   18 calls
       20 ms  CallTarget                                 1 calls
     1386 ms  Microsoft.TestPlatform.Build.Tasks.VSTestTask   1 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)

Expected Behaviour

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed LogDataConverterMapsLogLevelsToLogSeverities [22 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (None,Unknown) [12 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Trace,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Debug,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Information,Information) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Warning,Warning) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Error,Error) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Critical,Error) [< 1 ms]
  Passed LogDataConverterDetectsUnknownLogLevelMappings [< 1 ms]
  Passed LogStoreProviderEnumDefinedTest [196 ms]
  Passed LogStoreProviderEnumNotDefinedTest [< 1 ms]

Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.1466 Seconds
                   Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask". (TaskId:22)
12:58:07.009     1>Done building target "VSTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:5)
12:58:07.009     1>Target "GenerateCoverageResult: (TargetId:7)" in file "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.targets" from project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (target "GenerateCoverageResultAfterTest" depends on it):
                   Using "Coverlet.MSbuild.Tasks.CoverageResultTask" task from assembly "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.tasks.dll".
                   Task "Coverlet.MSbuild.Tasks.CoverageResultTask" (TaskId:23)
                     Task Parameter:ThresholdStat=minimum (TaskId:23)
                     Task Parameter:ThresholdType=line,branch,method (TaskId:23)
                     Task Parameter:InstrumenterState=/tmp/tmp0vqEQf.tmp (TaskId:23)
                     Task Parameter:OutputFormat=json (TaskId:23)
                     Task Parameter:Threshold=0 (TaskId:23)
                     Task Parameter:Output=/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json (TaskId:23)

Calculating coverage result...
                     [coverlet] Hit file '/tmp/COM.iX.Logging.Provider_6c1ec4a4-7989-466c-b501-b83278e7731f' deleted (TaskId:23)
  Generating report '/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json'

+-----------------------------+------+--------+--------+
| Module                      | Line | Branch | Method |
+-----------------------------+------+--------+--------+
| COM.iX.Logging.Provider | 80%  | 62.5%  | 66.66% |
+-----------------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 80%  | 62.5%  | 66.66% |
+---------+------+--------+--------+
| Average | 80%  | 62.5%  | 66.66% |
+---------+------+--------+--------+

                     Output Item(s): 
                         CoverletReport=
                             /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json
                                     Format=json (TaskId:23)
                   Done executing task "Coverlet.MSbuild.Tasks.CoverageResultTask". (TaskId:23)
12:58:07.261     1>Done building target "GenerateCoverageResult" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:7)
12:58:07.261     1>Target "GenerateCoverageResultAfterTest: (TargetId:8)" in file "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.targets" from project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (entry point):
12:58:07.261     1>Done building target "GenerateCoverageResultAfterTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:8)
12:58:07.262     1>Done Building Project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (VSTest target(s)).
Deferred Messages
12:58:07.266     0>
Detailed Build Summary
======================
     (TaskId:0)
                     
                     ============================== Build Hierarchy (IDs represent configurations) =====================================================
                     Id                  : Exclusive Time   Total Time   Path (Targets)
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     0                   : 2.658s           2.658s       /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj (VSTest)  (TaskId:0)
                     
                     ============================== Node Utilization (IDs represent configurations) ====================================================
                     Timestamp:            1        Duration   Cumulative
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     637508230846046671:   0        2.661s     2.661s ..................................................... (TaskId:0)
                     -----------------------------------------------------------------------------------------------------------------------------------
                     Utilization:          100.0    Average Utilization: 100.0 (TaskId:0)

Project Evaluation Performance Summary:
      350 ms  /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls

Project Performance Summary:
     2256 ms  /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls
               2256 ms  VSTest                                     1 calls

Target Performance Summary:
        0 ms  InstrumentModulesNoBuild                   1 calls
        0 ms  GenerateCoverageResultAfterTest            1 calls
        3 ms  ShowCallOfVSTestTaskWithParameter          1 calls
       12 ms  GenerateProgramFile                        1 calls
      252 ms  GenerateCoverageResult                     1 calls
      572 ms  InstrumentModules                          1 calls
     1414 ms  VSTest                                     1 calls

Task Performance Summary:
        2 ms  Message                                   18 calls
        8 ms  CallTarget                                 1 calls
      251 ms  Coverlet.MSbuild.Tasks.CoverageResultTask   1 calls
      567 ms  Coverlet.MSbuild.Tasks.InstrumentationTask   1 calls
     1399 ms  Microsoft.TestPlatform.Build.Tasks.VSTestTask   1 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)
@pankti11 pankti11 closed this as completed Mar 9, 2021
@pankti11 pankti11 reopened this Mar 9, 2021
@MarcoRossignoli
Copy link
Collaborator

hi @pankti11 sorry for the delay, very busy days, outside container does it work?

@MarcoRossignoli MarcoRossignoli added the waiting for customer Waiting for customer action label Mar 16, 2021
@pankti11
Copy link
Author

Yes. but the only difference is the outside container I do a build. However, inside the container, I don't do-build. However, the bin folder (dlls are present from the build outside the container) is present in the container. So when I do dotnet test with --nobuild I expect it to generate the code coverage. Maybe I am missing something.

@petli
Copy link
Collaborator

petli commented Mar 19, 2021

Do you copy the source code into the docker container in addition to the binaries? Coverlet skips projects with PDBs but no local source files as a heuristic to filter out libraries that shouldn't be part of the coverage. (See https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Instrumentation/Instrumenter.cs#L88)

@petli
Copy link
Collaborator

petli commented Mar 19, 2021

It may also be an issue that the path name of the directory containing the code in the docker image is different to the path name outside it, which means that the source files cannot be found even if they are copied into the image. You can try to enable verbose logging (see https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md) and see if it logs messages about not finding the source files.

@petli petli mentioned this issue Mar 19, 2021
Closed
@pankti11
Copy link
Author

So inside the container there is source file as well as the binaries. From the logs I could decipher that the Build Task which is responsible to generate the CodeCoverage is never runs. Hence, I concluded that when I do dotnet test --no-build, dotnet is not aware of the coverlet because I did not run dotnet build. When I do dotnet build inside container and then do dotnet test --no-build then the coverage is created by coverlet without any issue.

@TFTomSun
Copy link
Contributor

TFTomSun commented Jul 10, 2022 •
edited
Loading

@petli We are running into similar issues. Actually we want to build and pack the binaries in one job and consume the nuget packages in multiple test jobs. Would be optimal if we wouldn't have to map the sources in the test job. Is that a requirement?

Additionally we tried as a workaround to map the sources along with the test run, but even that doesn't work. The path to the sources is identically, but of course we didn't do a build in the test job. Is there some output expected relative to the sources which we need to transfer from our build job to the test jobs? (maybe some files in the obj folder? )

Can I somehow configure to instrument binaries with embedded pdbs, even if no sources for them are available?
We have the sources embdeded, along with the embedded pdb. Are embedded sources not supported?

btw. we are using coverlet.collector

@TFTomSun TFTomSun mentioned this issue Jul 10, 2022
Merged
@github-actions
Copy link

This issue is stale because it has been open for 3 months with no activity.

@github-actions github-actions bot added the stale label Sep 17, 2023
@github-actions GitHub Actions
Copy link

This issue was closed because it has been inactive for 9 months since being marked as stale.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale waiting for customer Waiting for customer action
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@petli @TFTomSun @MarcoRossignoli @pankti11