Skip to content

fix(data): honor HumanEval+ version when loading and hashing - #316

Open
JianYan11 wants to merge 1 commit into
evalplus:masterfrom
JianYan11:fix/humaneval-dataset-version
Open

JianYan11 wants to merge 1 commit into
evalplus:masterfrom
JianYan11:fix/humaneval-dataset-version

Conversation

@JianYan11

Copy link
Copy Markdown

Summary

Honor the existing version argument when loading and hashing HumanEval+.

_ready_human_eval_plus_path() resolves version, but then passes the constant HUMANEVAL_PLUS_VERSION to get_dataset_metadata(). Independently, get_human_eval_plus_hash() replaces its caller's version with "default". As a result, requesting an older release can silently read and hash the default release instead.

Forward the resolved/caller-supplied version in these two places, matching the existing MBPP+ implementation. Default-version behavior and HUMANEVAL_OVERRIDE_PATH precedence remain unchanged. No release data, scoring logic, cache format, or dependencies are changed.

Tests

Added offline tests using synthetic JSONL files in pytest's temporary directory. Only make_cache() is mocked: metadata/path construction, file loading, completeness checking, and hashing use the real implementations.

  • Verify default and explicit versions for regular, Mini, and NoExtreme paths, independently for the loader and hash helper.
  • Populate both requested and default caches with distinct content to detect silently selecting the wrong release.
  • Verify that an override path still wins without invoking cache/download setup.
  • Before the fix: 6 failed, 7 passed (wrong contents/hashes).
  • After the fix: 13 passed.
PYTHONPATH=. uv run --no-project --python 3.13 \
  --with datasets --with tempdir --with wget --with appdirs --with pytest -- \
  python -m pytest tests/test_humaneval_version.py -q

Black 22.6.0, isort 5.12.0 with the Black profile, and git diff --check pass. No benchmark dataset downloads, generated-code execution, model calls, or full-suite run.

AI assistance: implemented and tested with OpenAI Codex.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant