Skip to content

Unit tests aren't discovered in sub-folders #11969

Closed
Closed
Unit tests aren't discovered in sub-folders#11969
Assignees
karrtikr
Labels
bugIssue identified by VS Code Team member as probable buginfo-neededIssue requires more information from posterinvestigatingWe are looking into the cause of the issue

Description

@ItsDrike
ItsDrike (ItsDrike)
opened on May 24, 2020

Environment data

  • VS Code version: 1.45.1
  • Extension version (available under the Extensions sidebar): 2020.5.80290
  • OS and version: Arch Linux (kernel: linux 5.6.14.arch1-1)
  • Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.3 64-bit
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
  • Relevant/affected Python packages and their versions: unittest
  • Relevant/affected Python-related VS Code extensions and their versions: None
  • Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): True
  • Value of the python.languageServer setting: Microsoft

Actual behaviour

In main tests/ directory are some tests and some other files needed. Those work fine until a subdirectory is created, which contains 2 tests named test_decorators.py and test_pagination.py, this subdirectory also contains the __init__.py so that the tests can be recognized. After this subdirectory is created, the tests from main directory (tests/) disappear and the ones from subdirectory are not displayed either, a popup saying that no tests were discovered and asking me to configure the test framework (It is configured properly)

Once the __init__.py file is added and I rediscover the tests, this box pops up.
image

Expected behaviour

When I add the subdirectory to main tests/ directory and search for the tests, both the tests from my main directory and the subdirectory should be found.

Steps to reproduce:

  1. Create directory for tests in root of your project tests/
  2. Configure the framework to use unittests and recognise files labeled: test_*.py
  3. Add some tests in it and let the framework discover them [To this point everything is as expected]
  4. Add a subdirectory in your tests/ directory and place some other tests in it
  5. Run test discovery again
  6. No tests were found

Peek 2020-05-24 16-07

Logs

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging) 

[Extension Host] Info Python Extension: 2020-05-24 16:13:27: getActivatedEnvironmentVariables, Class name = E, completed in 1ms, has a truthy return value, Arg 1: <Uri:/home/itsdrike/Programming/Python/Projects/Discord-Bot>, Arg 2: undefined, Arg 3: undefined

[Extension Host] Info Python Extension: 2020-05-24 16:13:27: > ~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -c "import unittestloader = unittest.TestLoader()suites = loader.discover("./tests", pattern="test_*.py")print("start") #Don't remove this linefor suite in suites._tests:    for cls in suite._tests:        try:            for m in cls._tests:                print(m.id())        except:            pass"

[Extension Host] Info Python Extension: 2020-05-24 16:13:27: cwd: ~/Programming/Python/Projects/Discord-Bot

Activity

  1. ItsDrike (ItsDrike)
    added
    triage-neededNeeds assignment to the proper sub-team
    bugIssue identified by VS Code Team member as probable bug
    on May 24, 2020

  2. ItsDrike commented on May 26, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on May 26, 2020
    Author

    Any ideas why this is happening or what ca I do to prevent it?

  3. Karthik Nadig (karthiknadig)
    assigned
    Eric Snow (ericsnowcurrently)
    on May 26, 2020
  4. Karthik Nadig (karthiknadig)
    added
    triage
    on May 26, 2020
  5. ghost removed
    triage-neededNeeds assignment to the proper sub-team
    on May 26, 2020
  6. Eric Snow (ericsnowcurrently)
    changed the title [-]Unit tests aren't discovered in sub-floders[/-] [+]Unit tests aren't discovered in sub-folders[/+] on May 27, 2020

  7. ericsnowcurrently commented on May 27, 2020

    @ericsnowcurrently
    Eric Snow (ericsnowcurrently)
    on May 27, 2020

    ItsDrike (@ItsDrike), thanks for the report. Test discovery definitely shouldn't be behaving this way.

    What happens when you run ~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -m unittest discover -p 'test_*.py' -t '.' -s './tests'?

    (roughly equivalent to the command from the log, ~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -c 'import unittest; loader = unittest.TestLoader(); suites = loader.discover("./tests", pattern="test_*.py"); print("start"); for suite in suites._tests: for cls in suite._tests: for m in cls._tests: print(m.id())')

  8. Eric Snow (ericsnowcurrently)
    added
    info-neededIssue requires more information from poster
    on May 27, 2020

  9. ItsDrike commented on May 27, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on May 27, 2020
    Author

    When I run the first command (~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -m unittest discover -p 'test_*.py' -t '.' -s './tests') all tests are found as seen in gif below (you can ignore that log message)

    Peek 2020-05-27 21-27

    but your second command has raised a SyntaxError:
    image

  10. ericsnowcurrently commented on May 27, 2020

    @ericsnowcurrently
    Eric Snow (ericsnowcurrently)
    on May 27, 2020

    Yeah, that second command (from the log file) is actually multi-line:

    ~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -c '
import unittest
loader = unittest.TestLoader()
suites = loader.discover("./tests", pattern="test_*.py")
print("start")
for suite in suites._tests:
    for cls in suite._tests:
        for m in cls._tests:
            print(m.id())
'

    The big question is if the two command actually are equivalent. I'm thinking they actually aren't. The "top_level_dir" argument to loader.discover() wasn't passed in the logged script. It defaults to the start dir ("./tests"), which I expect makes the tests fail. The equivalent command would be ~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -m unittest discover -p 'test_*.py' -s './tests' (note the missing -t arg). What happens when you run that?

  11. ItsDrike commented on May 27, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on May 27, 2020
    Author

    Running it as multiline still produces and error, but now it's an AttributeError

    image

    Running the `~/.local/share/virtualenvs/Discord-Bot-j5RKKR3Y/bin/python -m unittest discover -p 'test_*.py' -s './tests'` command as suggested produced very odd output: 

    EEEEEEEEE
======================================================================
ERROR: test_names_dont_shadow (bot.cogs.test_cogs.CommandNameTests)
Names and aliases of commands should be unique.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/test_cogs.py", line 70, in test_names_dont_shadow
    for cmd in self.get_all_commands():
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/test_cogs.py", line 62, in get_all_commands
    for module in self.walk_modules():
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/test_cogs.py", line 40, in walk_modules
    yield importlib.import_module(module.name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/test_security.py", line 6, in <module>
    from bot.cogs import security
ImportError: cannot import name 'security' from 'bot.cogs' (/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/__init__.py)

======================================================================
ERROR: bot.cogs.test_security (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.cogs.test_security
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/test_security.py", line 6, in <module>
    from bot.cogs import security
ImportError: cannot import name 'security' from 'bot.cogs' (/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/cogs/__init__.py)


======================================================================
ERROR: bot.test_converters (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.test_converters
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/test_converters.py", line 9, in <module>
    from bot.utils.converters import (
ModuleNotFoundError: No module named 'bot.utils.converters'


======================================================================
ERROR: bot.test_decorators (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.test_decorators
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/test_decorators.py", line 5, in <module>
    from bot.decorators import InChannelCheckFailure, in_channel
ModuleNotFoundError: No module named 'bot.decorators'


======================================================================
ERROR: bot.test_pagination (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.test_pagination
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/test_pagination.py", line 3, in <module>
    from bot import pagination
ImportError: cannot import name 'pagination' from 'bot' (/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/__init__.py)


======================================================================
ERROR: bot.utils.test_checks (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.utils.test_checks
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/utils/test_checks.py", line 3, in <module>
    from bot.utils import checks
ImportError: cannot import name 'checks' from 'bot.utils' (/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/utils/__init__.py)


======================================================================
ERROR: bot.utils.test_time (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: bot.utils.test_time
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/utils/test_time.py", line 8, in <module>
    from bot.utils import time
ImportError: cannot import name 'time' from 'bot.utils' (/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/bot/utils/__init__.py)


======================================================================
ERROR: test_base (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_base
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/test_base.py", line 6, in <module>
    from tests.base import LoggingTestsMixin, _CaptureLogHandler
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/base.py", line 9, in <module>
    from tests import helpers
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/helpers.py", line 12, in <module>
    from bot.bot import Bot
ModuleNotFoundError: No module named 'bot.bot'


======================================================================
ERROR: test_helpers (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_helpers
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/test_helpers.py", line 7, in <module>
    from tests import helpers
  File "/home/itsdrike/Programming/Python/Projects/Discord-Bot/tests/helpers.py", line 12, in <module>
    from bot.bot import Bot
ModuleNotFoundError: No module named 'bot.bot'


----------------------------------------------------------------------
Ran 9 tests in 0.003s

FAILED (errors=9)

    Seems like all 9 test files were run, but and all of them resulted in an error since the required modules which were supposed to be tested weren't even found.

    Peek 2020-05-27 23-33

  12. ItsDrike commented on May 31, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on May 31, 2020
    Author

    So is it a bug or should I do something to fix it on my machine?

  13. ericsnowcurrently commented on Jun 1, 2020

    @ericsnowcurrently
    Eric Snow (ericsnowcurrently)
    on Jun 1, 2020

    Right. Those failures are what I would expect You should see similar output in the "Python test log" output panel. What do you see there?

  14. ItsDrike commented on Jun 1, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on Jun 1, 2020
    Author

    The initial output was without init, with __init__.py there, It only really says start
    Peek 2020-06-01 23-36

  15. 10 remaining items

  16. Karthik Nadig (karthiknadig)
    removed
    triage
    on Sep 15, 2020

  17. yhgu2000 commented on Oct 23, 2020

    @yhgu2000
    顾宇浩 (yhgu2000)
    on Oct 23, 2020

    Any progress on this?

    I have came back to this page for about ten times, but nothing changed.......
    This bug makes the unittest function of this plugin almost useless. I have already disable it in my projects........

  18. ItsDrike commented on Oct 24, 2020

    @ItsDrike
    ItsDrike (ItsDrike)
    on Oct 24, 2020
    Author

    Any progress on this?

    I have came back to this page for about ten times, but nothing changed.......
    This bug makes the unittest function of this plugin almost useless. I have already disable it in my projects........

    Yeah, unit-testing doesn't work at all, I think this should have higher priority than it does now since testing is a very important part of every bigger python project and it's quite rare to have all of the tests in the same directory, not detecting this automatically is a pretty big issue, why is it taking so long to fix?

    It's already been over 5 months since I've opened the issue and frankly, that's just unacceptable for an issue like this.
    I believe I've done an okay job documenting it and providing enough details about it although I can give you more details if you need them.

    I don't often push on issues being fixed, but considering it was almost half of a whole year that has already passed, I'd really appreciate it if you looked into this a little further Eric Snow (@ericsnowcurrently)

  19. Brett Cannon (brettcannon)
    assigned
    Kartik Raj (karrtikr)
    and unassigned
    Eric Snow (ericsnowcurrently)
    on Mar 23, 2021

  20. karrtikr commented on Apr 6, 2021

    @karrtikr
    Kartik Raj (karrtikr)
    on Apr 6, 2021

    Is this issue still happening with the latest version of the extension?

  21. Kartik Raj (karrtikr)
    added
    info-neededIssue requires more information from poster
    on Apr 6, 2021

  22. yhgu2000 commented on May 11, 2021

    @yhgu2000
    顾宇浩 (yhgu2000)
    on May 11, 2021

    Is this issue still happening with the latest version of the extension?

    Yes, it is.

    You can use this minimal reproductive project to test: unittest_bug_example_project.zip.

  23. Kartik Raj (karrtikr)
    removed
    info-neededIssue requires more information from poster
    on May 11, 2021

  24. karrtikr commented on Jul 28, 2021

    @karrtikr
    Kartik Raj (karrtikr)
    on Jul 28, 2021

    @AtomicGu I used your sample repo but tests are discovered just fine.
    image

    I see the following in Python Test log output channel,

    start
test_1.Test_NoImport.test_1_1
test_1.Test_NoImport.test_1_2
test_1_1 (test_1.Test_NoImport) ... ok
test_1_2 (test_1.Test_NoImport) ... ok
test_2 (unittest.loader._FailedTest) ... ERROR
NoneType: None

======================================================================
ERROR: test_2 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_2
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\unittest\loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\unittest\loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "c:\Users\karraj\OneDrive - Microsoft\Desktop\folders\unittest_bug_example_project\package\tests\test_2.py", line 3, in <module>
    from .. import app
ImportError: attempted relative import with no known parent package


----------------------------------------------------------------------
Ran 3 tests in 0.010s

FAILED (errors=1)

    Can you send the output?

  25. Kartik Raj (karrtikr)
    added
    info-neededIssue requires more information from poster
    on Jul 28, 2021

  26. karrtikr commented on Jul 28, 2021

    @karrtikr
    Kartik Raj (karrtikr)
    on Jul 28, 2021

    Note we're rewriting the whole testing code with #16769, this will probably be fixed.

  27. karrtikr commented on Sep 1, 2021

    @karrtikr
    Kartik Raj (karrtikr)
    on Sep 1, 2021

    Should be fixed with the new testing work. If not, please open a new issue and we'll be happy to look into it.

  28. Kartik Raj (karrtikr)
    closed this as completedon Sep 1, 2021
  29. github-actions
    locked as resolved and limited conversation to collaborators on Oct 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugIssue identified by VS Code Team member as probable buginfo-neededIssue requires more information from posterinvestigatingWe are looking into the cause of the issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions