Skip to content

Subsequent runs produce different results #20

Open
Open
Subsequent runs produce different results#20

Description

@KiraResari
KiraResari
opened on Mar 17, 2022

I ran Delphi Code Coverage multiple times on the same code base, and got different results for the coverage even though I didn't change anything about the code in between.

The Command I used to run this was: CodeCoverage -e PBeratungsapp_Tests.exe -m PBeratungsapp_Tests.map -uf test_results/script/dcov_units.lst -spf test_results/script/dcov_paths.lst -od test_results -lt -xml -html -v -ife

The first day I got this result every time:

+-----------+-----------+-----------+
|   Lines   |  Covered  | Covered % |
+-----------+-----------+-----------+
|      6991 |      3463 |      49 % |
+-----------+-----------+-----------+

Today, the first time I ran it, I got this result:

  +-----------+-----------+-----------+
  |   Lines   |  Covered  | Covered % |
  +-----------+-----------+-----------+
  |      6991 |       426 |       6 % |
  +-----------+-----------+-----------+

I then rebuilt the test project (even though nothing about the code had changed), and then I got this result:

  +-----------+-----------+-----------+
  |   Lines   |  Covered  | Covered % |
  +-----------+-----------+-----------+
  |      6991 |      5161 |      73 % |
  +-----------+-----------+-----------+

These fluctuations strike me as odd. Any idea as to why this might be happening?

Activity

  1. Laurensvanrun commented on Mar 17, 2022

    @Laurensvanrun
    Laurensvanrun
    on Mar 17, 2022
    Contributor

    This indeed is strange and I have never seens this behaviour. Are you sure all tests ran correctly in your situations and there were no exceptions in the tests? Are you tests clear unit tests, or integration tests (which, if they communicate with datbases/file systems can give different results/behavior in different situations)?

  2. KiraResari commented on Mar 18, 2022

    @KiraResari
    KiraResari
    on Mar 18, 2022
    Author

    Some of the tests are integration tests, so maybe that's the issue. However, I have no idea of how to check which tests were executed correctly if I run it with the above command line. Is there another way to do that?

    Also, if it were just the integration tests, I would expect a much smaller discrepancy than 6%, 49% and 73%

  3. MHumm commented on Dec 21, 2022

    @MHumm
    MHumm
    on Dec 21, 2022

    Which testing framework do you use?
    DUnit or DUnitX? If the former you might want to use the GUI runner, which lists all failed tests at the bottom and marks them in red in the tree. If you use the DUnitX one, the console out put should tell you which ones failed and I think I always need to hit enter to close that so it should stay open. If not, create a batch file in which you cann that exe file with the tests and put a pause statement directly beding it. It keeps the console window open until you hit a key so you should see which tests failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions