Skip to content

Implement walk() #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thatch merged 4 commits into from
Oct 29, 2020
Merged

Implement walk() #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0510a0b
Implement walk()
spencerkee Oct 26, 2020
00d34c1
Wrap docstring at 88 chars, add note about os.walk directory order
spencerkee Oct 27, 2020
b098125
Sort expected/result files since os.walk traverses directories arbitr…
spencerkee Oct 27, 2020
4877e2f
Fix Windows incompatibility with test filepaths containing /
spencerkee Oct 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
PYTHON?=python
SHELL := /bin/bash

@spencerkee spencerkee Oct 26, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set up a new environment, Windows Subsystem Linux for Ubuntu 20.04 LTS. I found it necessary to add this line because otherwise by default make uses /bin/sh as per https://www.gnu.org/software/make/manual/html_node/Choosing-the-Shell.html which causes make venv to fail with:

source .venv/bin/activate && make setup
/bin/sh: 1: source: not found
make: *** [Makefile:7: venv] Error 127

I'm not sure if you prefer users pass in the shell; if so I can remove this line, and then perhaps something to that nature should be added to the quick development instructions.

PYTHON?=/usr/bin/env python3

@spencerkee spencerkee Oct 26, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least in Ubuntu 20.04, by default python3 is installed to /usr/bin/python3 and /usr/bin/python is unset. Since this seems to be a python3 compatible project, I added this line.

SOURCES=tuber setup.py

.PHONY: venv
Expand All @@ -12,6 +13,7 @@ venv:

.PHONY: setup
setup:
python -m pip install --upgrade wheel

@spencerkee spencerkee Oct 26, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly wheel is not included by default with pip on a fresh install of Ubuntu 20.04 LTS.

python -m pip install -Ur requirements-dev.txt

.PHONY: test
Expand Down
46 changes: 45 additions & 1 deletion tuber/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import os
from fnmatch import fnmatch
from pathlib import Path
from typing import Optional, Union
from typing import Iterable, Optional, Union

# This covers 99% of Python projects. For simplicity, stop at the first one
# found.
Expand All @@ -12,6 +14,25 @@
"setup.py",
}

DEFAULT_INCLUDES = ["*.py"]

DEFAULT_EXCLUDES = [

@spencerkee spencerkee Oct 26, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not exactly sure how black's default excludes works. On first glance it seems like it's a substring match which is why I added the asterisk to both sides of the string. I should also add a permalink to black here.

"*.direnv*",
"*.eggs*",
"*.ext.py*",
"*.git*",
"*.hg*",
"*.mypy_cache*",
"*.nox*",
"*.tox*",
"*.venv*",
"*.svn*",
"*_build*",
"*buck-out*",
"*build*",
"*dist*",
]


class RootException(ValueError):
pass
Expand Down Expand Up @@ -41,5 +62,28 @@ def get_root(path: Optional[Union[Path, str]] = None) -> Path:
path = parent


def walk(
root: Path,
includes: Iterable[str] = DEFAULT_INCLUDES,
excludes: Iterable[str] = DEFAULT_EXCLUDES,
) -> Iterable[str]:
"""Generator that calls os.walk on a path and applies includes/excludes

walk() uses fnmatch so Unix shell-style wildcards e.g. *, ?, [1-9] are supported.
For more information on the wildcards see
https://docs.python.org/3/library/fnmatch.html.
Note, the patterns in includes/excludes are only applied to the filename itself,
not the directories in the path. Additionally os.walk traverses directories in
arbitrary order.
"""
for dirpath, _, filenames in os.walk(root):
for filename in filenames:
if not any([fnmatch(filename, pattern) for pattern in includes]):
continue
if any([fnmatch(filename, pattern) for pattern in excludes]):
continue
yield os.path.join(dirpath, filename)


if __name__ == "__main__": # pragma: no cover
print(get_root())
3 changes: 2 additions & 1 deletion tuber/tests/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from .get_root import GetRootTest
from .walk import WalkTest

__all__ = ["GetRootTest"]
__all__ = ["GetRootTest", "WalkTest"]
65 changes: 65 additions & 0 deletions tuber/tests/walk.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import os
import tempfile

@spencerkee spencerkee Oct 26, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with the use of volatile over tempfile for creating temporary directories, I can switch too volatile if you prefer.

import unittest
from pathlib import Path

from tuber import walk

# Each filepath is separated into a list so that os.path.join() can connect
# them in a platform agnostic way without a predefined separator.
INCLUDED_TEST_FILEPATHS = [
["test.py"],
["abc", "test2.py"],
["abc", "def", "test3.py"],
["ghi", "test4.py"],
]

EXCLUDED_TEST_FILEPATHS = [
[".direnv"],
[".eggs"],
["abc", ".ext.py"],
["abc", ".git"],
["abc", ".hg"],
["abc", ".mypy_cache"],
["abc", "def", ".nox"],
["abc", "def", ".tox"],
["abc", "def", ".venv"],
["abc", "def", ".svn"],
["ghi", "_build"],
["ghi", "buck-out"],
["ghi", "build"],
["ghi", "dist"],
]


class WalkTest(unittest.TestCase):
@staticmethod
def _make_empty_file(path: str) -> None:
basedir = os.path.dirname(path)
if not os.path.exists(basedir):
os.makedirs(basedir)
open(path, "a").close()

def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
for relative_test_filepath in EXCLUDED_TEST_FILEPATHS + INCLUDED_TEST_FILEPATHS:
absolute_test_filepath = os.path.join(
self.temp_dir.name, *relative_test_filepath
)
self._make_empty_file(absolute_test_filepath)

def tearDown(self) -> None:
self.temp_dir.cleanup()

def test_inclusions_and_exclusions(self) -> None:
temp_dir_path = Path(self.temp_dir.name)
result_paths = list(walk(temp_dir_path))
self.assertEqual(
sorted(result_paths),
sorted(
[
os.path.join(temp_dir_path, *test_file)
for test_file in INCLUDED_TEST_FILEPATHS
]
),
)