Repository navigation

Support user-defined benchmark suites. - #109

Merged

ericsnowcurrently merged 127 commits into

python:mainfrom

ericsnowcurrently:benchmark-management

Dec 8, 2021

Merged

Support user-defined benchmark suites.#109
ericsnowcurrently merged 127 commits into
python:mainfrom
ericsnowcurrently:benchmark-management

Commits on Oct 4, 2021
- benchmarks -> _benchmarks
  ericsnowcurrently
  committed
  509d93c
- Use a new API for pyperformance.benchmarks.
  ericsnowcurrently
  committed
  e16141d
- Deal with benchmark objects instead of names.
  ericsnowcurrently
  committed
  d2101b2
- Refactor select_benchmarks().
  ericsnowcurrently
  committed
  20b2c23
- Add and use the default manifest file.
  ericsnowcurrently
  committed
  20e8492
- Move run_perf_script() back.
  ericsnowcurrently
  committed
  4f74cb8
- Clean up benchmark/__init__.py.
  ericsnowcurrently
  committed
  3917d22
- Make the utils a package.
  ericsnowcurrently
  committed
  32e15ec
- Move each of the default benchmarks into its own directory.
  ericsnowcurrently
  committed
  b041e2e
- Make BenchmarkSpec.metafile as "secondary" attribute.
  ericsnowcurrently
  committed
  d79eba5
- Fix benchmark selection.
  ericsnowcurrently
  committed
  f30421d
- Fix the default benchmarks selection.
  ericsnowcurrently
  committed
  043836b
- Fix the run script filename.
  ericsnowcurrently
  committed
  6ba9603
- Run benchmarks from the metadata instead of hard-coded.
  ericsnowcurrently
  committed
  0f15e8e
- Fix the requirements.
  ericsnowcurrently
  committed
  91c27f9
- Pass "name" through to parse_pyproject_toml().
  ericsnowcurrently
  committed
  a4f97ad
- Leave a note about classifiers.
  ericsnowcurrently
  committed
  6d385bf
- Drop an unused file.
  ericsnowcurrently
  committed
  5308b29
- Load manifest and select benchmarks before running the command.
  ericsnowcurrently
  committed
  4c8a18f
- Fix Benchmark.__repr__().
  ericsnowcurrently
  committed
  0f63a54
- Fix a default arg in load_metadata().
  ericsnowcurrently
  committed
  c7a92f4
- Fix the packaging data.
  ericsnowcurrently
  committed
  c45ece7
- Support per-benchmark venvs in VirtualEnvironment.
  ericsnowcurrently
  committed
  e940e24
- Ignore pyproject.toml name only if provided.
  ericsnowcurrently
  committed
  0420e96
- Add requirements lock files to the benchmarks.
  ericsnowcurrently
  committed
  6bce386
- Make a venv for each benchmark instead of sharing one.
  ericsnowcurrently
  committed
  29cf228
- Support "libsdir" in metadata.
  ericsnowcurrently
  committed
  6cadadc
- Fix an error message.
  ericsnowcurrently
  committed
  3fb5187
- Use the default resolve() if the default manifest is explicit.
  ericsnowcurrently
  committed
  abf56e9
- Merge in the version properly.
  ericsnowcurrently
  committed
  4bf223a
- Preserve PYTHONPATH (with libsdir) when invoking pyperf via benchmark scripts.
  ericsnowcurrently
  committed
  9acbbe4
- Add a note about using an upstream lib for parsing pyproject.toml.
  ericsnowcurrently
  committed
  d06aa7b
- Use the full benchmark version rather than the canonicalized form.
  ericsnowcurrently
  committed
  e48d29b
- Add iter_clean_lines() to _utils.
  ericsnowcurrently
  committed
  31f6021
- Use a run ID when running benchmarks.
  ericsnowcurrently
  committed
  4b13645
- Finish implementing "pre" and "post" script support.
  ericsnowcurrently
  committed
  b045e9a
- Drop "pre" and "post" script support. (It isn't necessary.)
  ericsnowcurrently
  committed
  54b8ba0
- Use the correct venv name for each benchmark.
  ericsnowcurrently
  committed
  362bb6e
- Stop supporting "libsdir" in the benchmark metadata.
  ericsnowcurrently
  committed
  e4fc5d4
- Clean up run_perf_script() and related code.
  ericsnowcurrently
  committed
  9816103
- Add a missing import.
  ericsnowcurrently
  committed
  b7f90f3
- Drop accidental files.
  ericsnowcurrently
  committed
  7d42a40
- Install all dependencies when running tests.
  ericsnowcurrently
  committed
  bb599e9
- Temporarily get tests passing on Windows.
  ericsnowcurrently
  committed
  df30711
- Ignore stdlib_dir mismatch for now.
  ericsnowcurrently
  committed
  8d563f9
Commits on Nov 2, 2021
- Move requirements.txt into the data dir.
  ericsnowcurrently
  committed
  d05b07c
- Move the benchmarks to the data dir.
  ericsnowcurrently
  committed
  701c910
- Move _pythoninfo out of the _utils dir.
  ericsnowcurrently
  committed
  afafda6
- Move _pyproject_toml out of the _utils dir.
  ericsnowcurrently
  committed
  cf4679a
- Make _utils a single module.
  ericsnowcurrently
  committed
  d3bdaad
- Move benchmarks.* up to the top level.
  ericsnowcurrently
  committed
  8f0b3e3
- Move benchmark.* up to the top level.
  ericsnowcurrently
  committed
  3ce65a6
- Clean up the metadata files.
  ericsnowcurrently
  committed
  3479e27
- Do not import _manifest in itself.
  ericsnowcurrently
  committed
  35e0c7a
- Drop version and origin from the manifest.
  ericsnowcurrently
  committed
  0fe268a
- Add a project-level symlink to the default benchmarks.
  ericsnowcurrently
  committed
  698dec9
- Names starting with a digit.
  ericsnowcurrently
  committed
  f3c6c4b
- Drop the base metadata file.
  ericsnowcurrently
  committed
  0987842
- All extra project fields even if there is a metabase.
  ericsnowcurrently
  committed
  2c99d0f
- Fix a typo.
  ericsnowcurrently
  committed
  121a281
- Allow specifying the supported groups in the manifest.
  ericsnowcurrently
  committed
  0f9d202
- Add tags to all the benchmarks.
  ericsnowcurrently
  committed
  3d19edf
Commits on Nov 3, 2021
- Use the tags to get the groups in the default manifest.
  ericsnowcurrently
  committed
  7270c76
- Show the default group before the others.
  ericsnowcurrently
  committed
  efeb77e
- Drop an outdated comment.
  ericsnowcurrently
  committed
  3b3de1d
- Allow excluding benchmarks in a group.
  ericsnowcurrently
  committed
  8ceb990
- Finish _init_metadata().
  ericsnowcurrently
  committed
  0f10859
- metabase -> inherits
  ericsnowcurrently
  committed
  9e3124d
- Document the manifest and benchmark formats.
  ericsnowcurrently
  committed
  62f7c23
Commits on Nov 6, 2021
- Fix some typos.
  ericsnowcurrently
  committed
  05bb86f
- Print some diagnostic info on error.
  ericsnowcurrently
  committed
  318720f
- Fall back to metadata for version.
  ericsnowcurrently
  committed
  b51042b
- Add benchmarks to the default group instead of names.
  ericsnowcurrently
  committed
  5eeae8e
- Install the requirements, even if the venv already exists.
  ericsnowcurrently
  committed
  0dc395a
Commits on Nov 8, 2021
- Only re-install reqs for benchmark venvs.
  ericsnowcurrently
  committed
  6fb7403
Commits on Nov 9, 2021
- Support an "includes" section in the manifest.
  ericsnowcurrently
  committed
  4979d5b
- doc fixes
  ericsnowcurrently
  committed
  7f7b571
- "all" and "default" are always valid groups.
  ericsnowcurrently
  committed
  0addec2
- Do not import pyperformance._manifest unless already installed.
  ericsnowcurrently
  committed
  57e7070
- Ensure we run in a venv when needed.
  ericsnowcurrently
  committed
  c55d690
- Do not re-install the shared venv.
  ericsnowcurrently
  committed
  01697d5
- Only list the "all" and "default" groups once.
  ericsnowcurrently
  committed
  94ac28a
- Add the manifest to the "compile" config.
  ericsnowcurrently
  committed
  f4b09bb
- Adjust the stdlib_dir check.
  ericsnowcurrently
  committed
  b56b25a
- Be sure to set base_executable.
  ericsnowcurrently
  committed
  f8338ae
Commits on Nov 15, 2021
- Use the --venv opt to the "compile" command.
  ericsnowcurrently
  committed
  dd4597b
- Separate the logic for create vs. recreate.
  ericsnowcurrently
  committed
  e22eeb8
- Do not re-create the venv if already running in it.
  ericsnowcurrently
  committed
  6bb3a9e
- Ensure all requirements are always isntalled.
  ericsnowcurrently
  committed
  9c664ff
- Do not buffer stdout during tests.
  ericsnowcurrently
  committed
  91c09d3
- Fix a check.
  ericsnowcurrently
  committed
  7a33956
- Do not buffer stdout during tests.
  ericsnowcurrently
  committed
  62e2014
- Always switch to a venv if running out of the repo.
  ericsnowcurrently
  committed
  d7ea256
- Distinguish message from runtests.py.
  ericsnowcurrently
  committed
  8ed6fd5
- Pass values into VirtualEnvironment instead of the options object.
  ericsnowcurrently
  committed
  ce6d09e
- Do not buffer stdout during tests.
  ericsnowcurrently
  committed
  4048199
- Distinguish message from runtests.py.
  ericsnowcurrently
  committed
  df8cccf
- Print out the --venv option.
  ericsnowcurrently
  committed
  8719636
- Print out the --venv option.
  ericsnowcurrently
  committed
  3c5d7da
- Print out the --venv option.
  ericsnowcurrently
  committed
  14761b2
Commits on Nov 16, 2021
- Do not add args directly to the "venv" command.
  ericsnowcurrently
  committed
  b975c51
- Be explicit about "create".
  ericsnowcurrently
  committed
  062745b
- Drop debug messages.
  ericsnowcurrently
  committed
  d918429
- Resolve the manifest file in the compile config.
  ericsnowcurrently
  committed
  5842d48
- Add a "dryrun" mode for testing "compile".
  ericsnowcurrently
  committed
  f1f9db1
- Add BenchmarkManifest.show().
  ericsnowcurrently
  committed
  35166b7
- Use --manifest and --benchmarks when creating venv for "compile".
  ericsnowcurrently
  committed
  561d271
- Add the resolve_file() util.
  ericsnowcurrently
  committed
  6831508
- Resolve the manifest file in includes.
  ericsnowcurrently
  committed
  bb88341
- Default BenchmarkRevision._dryrun to False.
  ericsnowcurrently
  committed
  1e51f8e
- Set the default for --benchmarks manually.
  ericsnowcurrently
  committed
  ef231fe
- Allow the "venv" command to not install benchmark requirements.
  ericsnowcurrently
  committed
  648bd18
- Use <NONE> as a marker for "no benchmarks".
  ericsnowcurrently
  committed
  7bb8d94
- Require --benchmarks (or default) for some commands.
  ericsnowcurrently
  committed
  15fd559
- Do not always install the first benchmark venv.
  ericsnowcurrently
  committed
  f041503
- Separate creating venv from installing requirements.
  ericsnowcurrently
  committed
  452b541
- Only install per-benchmark requirements when running them.
  ericsnowcurrently
  committed
  b049a4a
- Print the benchmark number.
  ericsnowcurrently
  committed
  a083cd5
Commits on Nov 17, 2021
- Factor out Python.resolve_program().
  ericsnowcurrently
  committed
  7b18753
- Do not pass --benchmarks when creating venv for "compile".
  ericsnowcurrently
  committed
  c8a7789
- Skip a benchmark if its requirements could not be installed.
  ericsnowcurrently
  committed
  6f6df4d
- Set Python.program to None if the resolved path does not exist.
  ericsnowcurrently
  committed
  b379536
- Factor out resolve_python().
  ericsnowcurrently
  committed
  686a96d
- Add a blank line.
  ericsnowcurrently
  committed
  664a909
- Do not print a traceback for skipped benchmarks.
  ericsnowcurrently
  committed
  9899813
- Fix a typo.
  ericsnowcurrently
  committed
  da1b6c3
Commits on Dec 7, 2021
- Merge main.
  ericsnowcurrently
  committed
  09ffb6b

Commits

Commits on Oct 4, 2021

Commits on Nov 2, 2021

Commits on Nov 3, 2021

Commits on Nov 6, 2021

Commits on Nov 8, 2021

Commits on Nov 9, 2021

Commits on Nov 15, 2021

Commits on Nov 16, 2021

Commits on Nov 17, 2021

Commits on Dec 7, 2021