Repository navigation
Split up JUnit XML output per file #4233
Description
Hello! I've recently started running mypy with --junit-xml since our CI system supports parsing JUnit output (and it's been a really useful feature!).
Currently, mypy's JUnit output lumps together all of the error output into one "testcase", like so:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="1" time="0.826">
<testcase classname="mypy" file="mypy" line="1" name="mypy" time="0.826">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement
testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
</testsuite>Would anyone have any objection to breaking this output up into 1 testcase per file? I imagine it would look something like this:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="3" time="0.826">
<testcase classname="testmodule.bar" file="testmodule/bar.py" line="0" name="testmodule.bar">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement</failure>
</testcase>
<testcase classname="testmodule.foo" file="testmodule/foo.py" line="0" name="testmodule.foo">
<failure message="mypy produced messages">testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
<testcase classname="testmodule.file_with_no_issues" file="testmodule/file_with_no_issues.py" line="0" name="testmodule.file_with_no_issues"></testcase>
</testsuite>There are a few benefits to splitting it up by file:
- CI systems that support the JUnit format can nicely format each error message per file, making it easier for developers to see which files need fixing
- CI systems that collect and aggregate metrics about which test cases fail most often / how many tests are run can have more detailed information to play with
- We can account for files that don't have typing issues (like
testmodule/file_with_no_issues.pyin the example above).
As an alternative, I considered having each typing error surface as a failed test case, but that makes the total number of tests and the number of failing tests vary wildly, making it difficult for CI systems that try to aggregate test case data.
I would love to implement this, just wanted to make sure no one had any objections to this before starting on it!
For implementation, I'm thinking of pulling in https://github.com/kyrus/python-junit-xml to actually generate the XML.
Activity
gvanrossum commentedon Nov 10, 2017
kennydo commentedon Nov 10, 2017
I put the sample per-file JUnit XML file through our CI system (Circle CI) with
tests="3"in thetestsuite, but with thetestcasewith no errors removed, and the UI displayed "Your build ran 2 tests with 2 failures", not 3. I think other CI systems would also expect each test to have an associatedtestcase, so omitting them might not be an option.In general, I'd really like to keep the number of testcases equal to the number of typechecked files. There are a few benefits, like being able to correlate the change in number of files type-checked with the change in execution time. Also, we can see when a previously failing file type later checks successfully (instead of having that file blink into and out of existince in the test suite). I think adding these successful testcases to the output XML won't be excessively large (the JUnit XML from our is also pretty massive).
Adding that newline sounds good to me!
gvanrossum commentedon Nov 11, 2017
dragongame commentedon May 25, 2018
Has any progress been made on this?
What is missing for this to be included?
gvanrossum commentedon May 25, 2018
Well there's a PR out but there are some questions raised in code review that haven't been answered AFAICT. Please follow up there if you have questions or feedback.
gvanrossum commentedon May 30, 2018
So, before you spend more time on my feedback, I showed the result to some coworkers and so far the only comment I got was "Hmm, I like it being more condensed myself. I wouldn't find that any more readable, just more clicks to see all the errors." I kind of sympathize. Note that this was on a diff where there happened to be errors in about 30 different files. Maybe if you really want this it could be enabled by a separate flag???
kennydo commentedon May 31, 2018
With files spread over 30 different files, having to do all that clicking does sound tedious.
I'd be OK with having a separate flag for this, since the utility of this change is pretty dependent on each developer/codebase's workflow. For our codebase, we're still incrementally adding typing, which is why we care about how many files successfully typechecked. We also generally run
mypylocally a few times on our changed files before CI is run on our branches, so the number of failing files in CI would drop off quickly.gvanrossum commentedon May 31, 2018
Honestly I'm torn. We already have too many flags. And in our (internal) CI the same problem occurs for failing tests -- if you have a lot of failing tests you must click on each one separately. Let's just proceed.
jisaacstone commentedon Apr 4, 2019
Yeah honestly having the whole thing be a single blob is less than ideal. I think most people would expect the report to have the correct metadata such as line number, file, etc.
However, since the output of mypy is nearly identical go flake8 the tools for flake8 such as
flake8-junit-reportpretty much just work with the mypy output. Anyone who desires a quick solution can just do that for now.But I'm more than happy to get the --junit-xml flag to work the same way if there is a consensus that it is what people want?
gvanrossum commentedon Apr 4, 2019
Let’s close this issue. I think it would be better to leave the xml generation to a separate tool.
dundee commentedon Feb 22, 2020
I just created simple tool for this task: https://pypi.org/project/mypy2junit/
Feedback welcomed!
Mattwmaster58 commentedon Mar 2, 2020
@dundee mypy can already generate JUnit XML, it's just that it isn't split up per file
dundee commentedon Mar 2, 2020
@Mattwmaster58 Yes, this is why I created this tool.
RJPercival commentedon Feb 8, 2024
This ended up being implemented by #16388.