test_plan.md

Test plan

Environment

OS: XXX (Windows, macOS, latest Ubuntu LTS)
- Shell: XXX (Command Prompt, PowerShell, bash, fish)
Python
- Distribution: XXX (CPython, miniconda)
- Version: XXX (2.7, latest 3.x)
VS Code: XXX (Insiders)

Tests

ALWAYS:

Check the Output window under Python for logged errors
Have Developer Tools open to detect any errors
Consider running the tests in a multi-folder workspace
Focus on in-development features (i.e. experimental debugger and language server)

Scenarios

Environment

Interpreters

Terminal

Sample file:

import requests
request = requests.get("https://drive.google.com/uc?export=download&id=1_9On2-nsBQIw3JiY43sWbrF8EjrqrR4U")
with open("survey2017.zip", "wb") as file:
    file.write(request.content)
import zipfile
with zipfile.ZipFile('survey2017.zip') as zip:
    zip.extractall('survey2017')
import shutil, os
shutil.move('survey2017/survey_results_public.csv','survey2017.csv')
shutil.rmtree('survey2017')
os.remove('survey2017.zip')

Shift+Enter to send selected code in sample file to terminal works

Virtual environments

ALWAYS:

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere
SPAM='hello ${WHO}'

ALWAYS:

Make sure to use Reload Window between tests to reset your environment
Note that environment files only apply under the debugger and Jedi
Environment variables in a .env file are exposed when running under the debugger
"python.envFile" allows for specifying an environment file manually (e.g. Jedi picks up PYTHONPATH changes)
envFile in a launch.json configuration works
simple variable substitution works

Debugging

pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS:

Check under the Problems tab to see e.g. if a linter is raising errors

Language server

LS is downloaded using HTTP (no SSL) when the "http.proxyStrictSSL" setting is false
Installing requests in virtual environment is detected
- Import of requests without package installed is flagged as unresolved
- Create a virtual environment
- Install requests into the virtual environment

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

Installation via the prompt installs Pylint as appropriate
- Uses --user for system-install of Python
- Installs into a virtual environment environment directly
Pylint works
"python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
The existence of a pylintrc file turns off the default rules

Other linters

Note:

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

"python.autoComplete.extraPaths" works
"python.autocomplete.addBrackets": true causes auto-completion of functions to append ()
Auto-completions works

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
import os,sys;
def foo():pass

Refactoring

Extract Variable works
- You are prompted to install rope if it is not already available
Extract method works
- You are prompted to install rope if it is not already available
Sort Imports works

Debugging

Unit testing

`unittest`

import unittest

MODULE_SETUP = False


def setUpModule():
    global MODULE_SETUP
    MODULE_SETUP = True


class PassingSetupTests(unittest.TestCase):
    CLASS_SETUP = False
    METHOD_SETUP = False

    @classmethod
    def setUpClass(cls):
        cls.CLASS_SETUP = True

    def setUp(self):
        self.METHOD_SETUP = True

    def test_setup(self):
        self.assertTrue(MODULE_SETUP)
        self.assertTrue(self.CLASS_SETUP)
        self.assertTrue(self.METHOD_SETUP)


class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

Run All Unit Tests triggers the prompt to configure the test runner
Tests are discovered (as shown by code lenses on each test)
- Code lens for a class runs all tests for that class
- Code lens for a method runs just that test
  - Run Test works
  - Debug Test works
  - Module/suite setup methods are also run (run the test_setup method to verify)
while debugging tests, an uncaught exception in a test does not cause ptvsd to raise SystemExit

`pytest`

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

Run All Unit Tests triggers the prompt to configure the test runner
- pytest gets installed
Tests are discovered (as shown by code lenses on each test)
- Run Test works
- Debug Test works
A Diagnostic is shown in the problems pane for each failed/skipped test
- The Diagnostics are organized according to the file the test was executed from (not necessarily the file it was defined in)
- The appropriate DiagnosticRelatedInformation is shown for each Diagnostic
- The DiagnosticRelatedInformation reflects the traceback for the test

`nose`

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

Run All Unit Tests triggers the prompt to configure the test runner
- Nose gets installed
Tests are discovered (as shown by code lenses on each test)
- Run Test works
- Debug Test works

General

Data Science

P0 Test Scenarios

Start and connect to local Jupyter server
1. Open the file src/test/datascience/manualTestFiles/manualTestFile.py in VSCode
2. At the top of the file it will list the things that you need installed in your Python environment
3. On the first cell click Run Below
4. Interactive Window should open, show connection information, and execute cells
5. The first thing in the window should have a line like this: Jupyter Server URI: http://localhost:[port number]/?token=[token value]
Verify basic outputs
1. Run all the cells in manualTestFile.py
2. Check to make sure that no outputs have errors
3. Verify that graphs and progress bars are shown
Verify export / import
1. With the results from Start and connect to local server open click the Export as Jupyter Notebook button in the Interactive Window
2. Choose a file location and save the generated .ipynb file
3. When the prompt comes up in the lower right choose to open the file in the browser
4. The file should open in the web browser and contain the output from the Interactive Window
5. In VSCode open up the exported .ipynb file in the editor, when the prompt for Do you want to import the Jupyter Notebook into Python code? appears click import
6. The imported file should match the original python file
Verify text entry
1. In the Interactive Window type in some new code print('testing') and submit it to the Interactive Windows
2. Verify the output from what you added
Verify dark and light main themes
1. Repeat the Start and connect to local server and Verify basic outputs steps using Default Dark+ and Default Light+ themes
Verify Variable Explorer
1. After manualTestFile.py has been run drop down the Variables section at the top of the Interactive Window
2. In the Variables list there should be an entry for all variables created. These variables might change as more is added to manualTestFile.py.
3. Check that variables have expected values. They will be truncated for longer items
4. Sort the list ascending and descending by Type. Also sort the list ascending and descending by Count. Values like (X, Y) use the first X value for Count sort ordering
5. Check that list, Series, ndarray, and DataFrame types have a button to "Show variable in data viewer" on the right
6. In the Interactive Window input box add a new variable. Verify that it is added into the Variable Explorer
Verify Data Explorer
1. From the listed types in the Variable explorer open up the Data Viewer by clicking the button or double clicking the row
2. Inspect the data in the Data Viewer for the expected values [ ] Verify Sorting and Filtering
  1. Open up the myDataFrame item
  2. Sort the name column ascending and descending
  3. Sort one of the numerical columns ascending and descending
  4. Click the Filter Rows button
  5. In the name filter box input 'a' to filter to just name with an a in them
  6. In one of the numerical columns input a number 1 - 9 to filter to just that column
3. Open the myList variable in the explorer
4. Make sure that you can scroll all the way to the end of the entries

P1 Test Scenarios

P2 Test Scenarios

Directory change
- Verify directory change in export
  1. Follow the previous steps for export, but export the ipynb to a directory outside of the current workspace
  2. Open the file in the browser, you should get an initial cell added to change directory back to your workspace directory
- Verify directory change in import
  1. Follow the previous steps for import, but import an ipynb that is located outside of your current workspace
  2. Open the file in the editor. There should be python code at the start to change directory to the previous location of the .ipynb file
Interactive Window input history history
1. Start up an Interactive Window session
2. Input several lines into the Interactive Window terminal
3. Press up to verify that those previously entered lines show in the Interactive Window terminal history
Extra themes
1. Try several of the themes that come with VSCode that are not the default Dark+ and Light+

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test plan

Environment

Tests

Environment

Interpreters

Terminal

Virtual environments

Environment files

Debugging

Linting

Language server

Pylint/default linting

Other linters

Editing

IntelliSense

Formatting

Refactoring

Debugging

Unit testing

`unittest`

`pytest`

`nose`

General

Data Science

P0 Test Scenarios

P1 Test Scenarios

P2 Test Scenarios

FilesExpand file tree

test_plan.md

Latest commit

History

test_plan.md

File metadata and controls

Test plan

Environment

Tests

Interpreters

Terminal

Virtual environments

Language server

Pylint/default linting

Other linters

General

P0 Test Scenarios

P1 Test Scenarios

P2 Test Scenarios