If multiple tests have the same name, only one is run

Issue Type: Bug

Here is a sample test script in which the same test is executed with different configurations.

import unittest

class Foo(unittest.TestCase):
    config = "Hello"
    def test_Print(self):
        print(self.config)

class FooConfigured(Foo):
    config = "World"

if __name__ == '__main__':
    unittest.main()

If I try to run all tests in the file, only FooConfigured is executed. If I try to run the Foo class as a whole, the same thing happens. I have to explicitly run Foo.test_Print or it will never be executed.

In my real world project there are 15 test methods, and the configuration required is picking a version of a dependency to run against.

Extension version: 2019.11.50794
VS Code version: Code 1.40.1 (8795a9889db74563ddd43eb0a897a2384129a619, 2019-11-13T16:49:30.162Z)
OS version: Linux x64 4.15.0-70-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (4 x 2531)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: disabled_software skia_renderer: disabled_off surface_control: disabled_off surface_synchronization: enabled_on video_decode: unavailable_off viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off webgl: enabled webgl2: enabled
Load (avg)	1, 1, 0
Memory (System)	11.46GB (7.43GB free)
Process Argv	--no-sandbox --unity-launch
Screen Reader	no
VM	0%

Tom Edwards (@Artfunkel)

Thanks for getting in touch with us on this, Tom Edwards (@Artfunkel). I was not able to reproduce the problem using the example code you provided. Instead, both tests were discovered and both were run. I tried it with both unittest and pytest. Here are some questions to help identify the problem:

what do you have in your workspace settings.json?
how are you running the tests?
what do you see in the test explorer?
do you see code lenses?
what do you see in the "Python Test Log" output panel?

Thanks!

I can reproduce this issue with a clean project, but the settings.json of my real project says this:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./Tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": true,
    "python.pythonPath": "/usr/bin/python3.7"
}

I'm running tests from the standard VS Code "Tests" panel by clicking on the green "Run" icons.

I see all of my tests listed correctly.

I see code lens on both test classes, but not on the individual methods.

After clicking the green "Run" icon on the .py file, I see this in the output log:

test_tests.Foo.test_Print
test_tests.FooConfigured.test_Print
test_Print (test_tests.FooConfigured) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OKWorld

I was able to reproduce the problem, but only in one certain case. If I run tests for the file (in test explorer or by command) then only the one test is run, even though both are discovered. If I "run all tests" then both tests get run. (I got the same behavior under unittest and pytest.)

We will look into this and work on a fix. Sorry for the inconvenience!

As to code lenses, there is no function to add one to in FooConfigured, so that is the expected behavior.

Possible causes:

the discovered test is not getting recorded properly (relative to the file)
the code that actually runs the tests for a file is making an invalid assumption and not handling the case (same file, same test name, same bound class, different discovered class) properly

This is fixed in our upcoming test provider.

Navigation Menu

Repository navigation

If multiple tests have the same name, only one is run #8761

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

If multiple tests have the same name, only one is run #8761

Description

Activity

ericsnowcurrently commented on Nov 25, 2019

Artfunkel commented on Nov 26, 2019

ericsnowcurrently commented on Nov 27, 2019

ericsnowcurrently commented on Nov 27, 2019

ericsnowcurrently commented on Nov 27, 2019

brettcannon commented on Jul 29, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions