Skip to content

When a test case name is prefix of another the wrong tests are run #7568

Closed
Closed
When a test case name is prefix of another the wrong tests are run#7568
Labels
area-testingbugIssue identified by VS Code Team member as probable bug

Description

@memeplex
memeplex
opened on Sep 24, 2019

Environment data

  • VS Code version: 1.38.1
  • Extension version: 2019.9.34911
  • OS and version: Ubuntu 19.04
  • Python version: 3.7.3
  • Type of virtual environment used: N/A
  • Relevant/affected Python packages and their versions: unittest
  • Jedi or Language Server?: Jedi

Expected behaviour

Running tests by clicking the name of a TestCase in the test view will run the tests in that TestCase.

Actual behaviour

When the names of two TestCases share the same prefix, only the tests for the one with the longer name are run, as shown in the following screenshot:

image

Steps to reproduce:

  1. Create a folder with contents:
<folder>/
     tests/
           __init__.py
           test_stuff.py
  1. Copy the code in the screenshot above into test_stuff.py.
  2. Open the test view as in the screenshot above.
  3. Double click on TestStuff.
  4. Double click on TestStuffBuilder.
  5. Only test_b is run.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

User belongs to experiment group 'ShowExtensionSurveyPrompt - control'
Starting Jedi Python language engine.
> conda --version
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> /bin/python3 -c "import sys;print(sys.executable)"
> conda info --json
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/interpreterInfo.py
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> conda info --json
> conda --version
> /bin/python3 -c "import sys;print(sys.prefix)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.prefix)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.executable)"
cwd: /tmp/test
> /bin/python3 -c "import sys;print(sys.executable)"
cwd: /tmp/test
> /bin/python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /tmp/test
> /bin/python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
cwd: /tmp/test
> /bin/python3 -m site --user-site
cwd: /tmp/test
> /bin/python3 -m site --user-site
cwd: /tmp/test
> /bin/python3 completion.py
cwd: ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles
> /bin/python3 completion.py
cwd: ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/test_stuff.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/test_stuff.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
> /bin/python3 -m flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s /tmp/test/tests/__init__.py
cwd: /tmp/test
##########Linting Output - flake8##########
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="test_*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=45175 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=45175 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39933 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39933 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=43647 -ttests.TestStuff.test_a --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=43647 -ttests.TestStuff.test_a --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33905 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33905 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=37247 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=37247 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39113 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=39113 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33985 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=33985 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=46605 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=46605 -ttests.TestStuff --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=41825 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test
> /bin/python3 ~/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=test_*.py --uvInt=2 --result-port=41825 -ttests.TestStuffBuilder --testFile=/tmp/test/tests.py
cwd: /tmp/test

Test Logs

start
start
tests.TestStuff.test_a
tests.TestStuffBuilder.test_b
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_a (tests.TestStuff) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
test_b (tests.TestStuffBuilder) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Activity

  1. memeplex
    added
    triage-neededNeeds assignment to the proper sub-team
    bugIssue identified by VS Code Team member as probable bug
    on Sep 24, 2019
  2. Kartik Raj (karrtikr)
    added
    triage
    on Sep 24, 2019
  3. Kartik Raj (karrtikr)
    self-assigned this
    on Sep 24, 2019
  4. ghost removed
    triage-neededNeeds assignment to the proper sub-team
    on Sep 24, 2019
  5. Kartik Raj (karrtikr)
    added
    info-neededIssue requires more information from poster
    on Sep 24, 2019

  6. karrtikr commented on Sep 24, 2019

    @karrtikr
    Kartik Raj (karrtikr)
    on Sep 24, 2019

    I am able to confirm this, thanks! I assume you're using unittest framework.

    Double click on TestStuff.

    I am confused here😕 AFAICS double clicking a item in the test explorer does nothing but opens the file. Are you referring to the play icon next to each test item? I used that to run the TestCase.

  7. Kartik Raj (karrtikr)
    added
    area-testing
    needs PR
    reason-preexisting
    and removed
    info-neededIssue requires more information from poster
    triage
    on Sep 24, 2019
  8. Kartik Raj (karrtikr)
    removed their assignment
    on Sep 24, 2019

  9. memeplex commented on Sep 27, 2019

    @memeplex
    memeplex
    on Sep 27, 2019
    Author

    I am confused hereconfused AFAICS double clicking a item in the test explorer

    You're right, sorry, I should have said "clicking on the arrow/play icon".

    I assume you're using unittest framework.

    Yes, just the builtin unittest framework.

  10. memeplex commented on Sep 29, 2019

    @memeplex
    memeplex
    on Sep 29, 2019
    Author

    Kartik Raj (@karrtikr) you added the "needs PR" label, does it mean that you're expecting a PR from me or is it more like an internal label?

  11. karrtikr commented on Sep 29, 2019

    @karrtikr
    Kartik Raj (karrtikr)
    on Sep 29, 2019

    It just signifies that the issue is confirmed and is ready to be worked on. You or any other external contributer are welcome to submit PRs but you're not expected to.

  12. memeplex commented on Oct 8, 2019

    @memeplex
    memeplex
    on Oct 8, 2019
    Author

    Kartik Raj (@karrtikr) I'm finding the issue is worse than I described. Many times only the first test is run even if no test is a prefix of another one:

    image

    I'm attaching a very simple sample workspace for you to try.

    proj.zip

  13. memeplex commented on Oct 8, 2019

    @memeplex
    memeplex
    on Oct 8, 2019
    Author

    I think PR #7783 is a simple fix for both #7568 and #4567, which are both marked as needs PR. It requires one reviewer.

  14. karrtikr commented on Nov 7, 2019

    @karrtikr
    Kartik Raj (karrtikr)
    on Nov 7, 2019

    memeplex Sorry for the late response, was away for a while. PR #7783 has been merged. Can you use Switch to insiders weekly command to install the insiders build of the extension, and see if it fixes the issue?

  15. Luciana Abud (luabud)
    closed this as completedon Dec 8, 2020
  16. ghost removed
    needs PR
    on Dec 8, 2020
  17. github-actions
    locked as resolved and limited conversation to collaborators on Dec 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-testingbugIssue identified by VS Code Team member as probable bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions