Plans for mypy.myunit #908

Since driver is now merged, mypy.myunit is a package instead of a module; however all the logic is still in __init__.py and has irritating limitations.

Note that in these plans I am not yet making myunit and the driver aware of each other, just laying out a framework.

• Document which parts of myunit are actually used from elsewhere.
• Remove the unnecessary use of Any.
• Split "collect tests" logic from "run tests" logic, add -l to only collect and print.
• Split "collect results" logic from "print results" logic.
• Allow fetching a single test by name (see below) instead of collecting everything and then filtering even when there are no wildcards.
• Use fnmatch instead of hand-written glob functions (likely via regex conversion, so we can add a command-line option to take a regex directly).
• Add tests for myunit itself.
• Move assertions so you can do from mypy.myunit.assertions import *
• Add decorators in mypy.myunit.annotations, see https://gist.github.com/o11c/ef8f0886d5967dfebc3d
• Also add an decorator for "this is an assert_foo function" to replace the current traceback-cleaning logic, which is wrong if something goes wrong in the assert function itself.
• Add a check that some assert_foo function is actually run for every testcase.
• Record results of a test run to file so that diffs can be displayed from previous runs.
• Move mutable globals to mypy.myunit.ick
• Move main function and related logic to mypy.myunit.main
• Allow free functions to be used as tests instead of requiring them to be wrapped in a class.
• Support opting out of the tempdir logic.
• Support fixtures by looking up (name.strip('_'), annotated_type) in current module and in mypy.myunit.fixtures (use this for tempdir maybe?)
• Change data-driven testcases from overriding cases to overriding init, likely in a base class that takes care of calling parse_tests.
• Support a "keep going" mode of assertion failure.

Anyway, the big change is global test naming.

New test names have more dots, and will look like mypy.test.testlex.LexerSuite.test_empty. It is not always as simple as getattr - modules need to be imported, classes need to be instantiated, data-driven cases need to be parsed, etc.

The first component name will always be a module, and most likely always be just 'mypy' depending on what is done about #724

I introduce two new classes, TestFactory and Test, to avoid refactoring the responsibilities of Suite and TestCase. There will be a bridge of course, or maybe the old classes will just be subclasses of the new ones. Possibly the old classes will go away entirely.

API will look something like:

TestOrTestFactory = Union['Test', 'TestFactory']

class Test:
    name = ''
    # Basically just TestCase without all the cruft,
    # but also stores the annotations.
    # It is now forbidden to create your own subclasses, use fixtures instead.

class TestFactory:
    name = ''
    @abstractmethod
    def list_immediate_children(self) -> Iterable[str]: ...
    @abstractmethod
    def get_immediate_child(self, name: str) -> TestOrTestFactory: ...

    @staticmethod
    def list_recursive_children(name: str) -> Iterable[str]: ...
    @staticmethod
    def get_recursive_child(name: str) -> TestOrTestFactory: ...

class ModuleTestFactory(TestFactory):
    def __init__(self, module: types.ModuleType) -> None: ...
    # implementation uses `getattr`, `__path__`, and `__import__`
    # to find both contained modules (but not imported ones from elsewhere!)
    # and test objects (functions, classes, or instances) from the current module.

class SuiteTestFactory(TestFactory):
    def __init__(self, suite: Suite) -> None: ...
    # implementation uses `suite.cases()`, now restricted to only return `TestCase`s.
    # (The only remaining suite with suites as children is the temporary `ListSuite`).

Bikeshed: what do we call the module in which Suite, TestCase, TestFactory, and Test reside? Same module or separate ones?