fix(scripts): test_sandbox_write.sh and sandbox_write_live.py honour direct auth profile#179
Conversation
When YANDEX_DIRECT_TOKEN and YANDEX_DIRECT_LOGIN are not in env, fall back to the OAuth profile resolved by direct_cli.auth.get_credentials (profile → env → .env → vault priority). Matches the existing behaviour of scripts/test_safe_commands.sh. Two layers fixed symmetrically: - scripts/test_sandbox_write.sh: probe `direct auth status` for `has_token=yes` (the command always exits 0, so grep the marker), then inject resolved token+login via eval so subprocesses see them. - scripts/sandbox_write_live.py:validate_environment(): defensive fallback to get_credentials() for direct python invocations bypassing the shell wrapper. CI is not affected — these runners are manual-only, not invoked from .github/workflows. Closes #178. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…is unset Companion fix to #179 (auth-profile fallback in WRITE_SANDBOX runners). Previously _credentials() silently skipped all v4_live_read tests for profile-only users — manual pytest -m v4_live_read showed no signal. Now: when env vars are missing, try direct_cli.auth.get_credentials() to resolve the profile (matches the priority used by every CLI subcommand). Skip message also tells the user both options. Refs #178. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
Restores parity with scripts/test_safe_commands.sh by letting the WRITE_SANDBOX runners fall back to an active direct auth login profile when YANDEX_DIRECT_TOKEN/YANDEX_DIRECT_LOGIN are not in the environment, so OAuth-profile users can run the sandbox-write smoke tests without manually exporting raw credentials.
Changes:
scripts/test_sandbox_write.sh: when env vars are missing, probedirect auth statusforhas_token=yesandeval-export credentials resolved viadirect_cli.auth.get_credentialsso child Python anddirect --sandboxinvocations inherit them.scripts/sandbox_write_live.py::validate_environment(): defensive fallback toget_credentials(None, None)for direct Python entry, mutatingos.environso spawned subprocesses see the credentials; emits a clear error if no profile resolves either.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| scripts/test_sandbox_write.sh | Shell-level fallback to auth profile via direct auth status + eval-export of resolved token/login. |
| scripts/sandbox_write_live.py | validate_environment() now falls back to get_credentials() and populates os.environ for subprocesses. |
Comments suppressed due to low confidence (1)
scripts/test_sandbox_write.sh:28
get_credentials(None, None)returnsTuple[str, Optional[str]]—logincan legitimately beNone(seedirect_cli/auth.py:600-702). When that happens this block printsexport YANDEX_DIRECT_LOGIN=None, exporting the literal string"None"as the login, which the downstreamdirect --sandbox …calls will then attempt to use. Validate that bothtokenandloginare truthy before printing the export lines and exit with a clear error otherwise, mirroring the guard already present insandbox_write_live.py:validate_environment().
eval "$(python3 -c '
from direct_cli.auth import get_credentials
token, login = get_credentials(None, None)
print(f"export YANDEX_DIRECT_TOKEN={token}")
print(f"export YANDEX_DIRECT_LOGIN={login}")
')"
| from direct_cli.auth import get_credentials | ||
| token, login = get_credentials(None, None) | ||
| print(f"export YANDEX_DIRECT_TOKEN={token}") | ||
| print(f"export YANDEX_DIRECT_LOGIN={login}") |
There was a problem hiding this comment.
Addressed in ab927ea — shlex.quote() applied to both token and login, plus an explicit if not token or not login: sys.exit(...) guard with proper subshell-exit propagation via if ! resolved=$(...). Latest Claude review on the PR confirms both findings resolved.
…via CLI Addresses Codex review finding #2 on #179: write-mutating tests were inheriting the v4_live_read file-level marker, breaking the trust boundary between read-only and write test suites. - Removes test_v4_live_wordstat_lifecycle_contract_opt_in_write and test_v4_live_forecast_lifecycle_contract_opt_in_write from tests/test_v4_live_contracts.py (which is correctly marked v4_live_read). - Adds test_live_draft_v4wordstat_lifecycle and test_live_draft_v4forecast_lifecycle to tests/test_integration_live_write.py under the integration_live_write marker. - Rewrites them to use the CLI (via _invoke_live / CliRunner) instead of native call_v4, matching the convention of every other live-write test in that file. - Drops in-test YANDEX_DIRECT_LIVE_WRITE checks — the file-level skipif already enforces the env gate. Verified: pytest --collect-only -m v4_live_read tests/test_v4_live_contracts.py → 9 read-only tests (no lifecycle probes). pytest --collect-only -m integration_live_write tests/test_integration_live_write.py → both v4 lifecycle tests included. env -u YANDEX_DIRECT_LIVE_WRITE pytest <…lifecycle…> → SKIPPED via file-level skipif. Codex review finding #1 (bare `pytest` hits live API when profile is active) is intentionally deferred: project decision is that v4 live read tests should run whenever a profile is available. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
Triage of the Codex adversarial review: Finding 2 — Mutating probes inherit the read-only Finding 1 — Active auth profile turns default test runs into live API calls — declined as an intentional project decision. The maintainer chose that bare `pytest` with an active `direct auth login` should hit Yandex Direct read endpoints. No `addopts` exclusion will be added. Verification: ```bash This PR (#179) itself is unchanged — it only touches `scripts/`. The marker / location fix lives entirely in #177. |
|
@claude review. Focus on critical issues: bugs, security vulnerabilities, logical errors, data loss risks, performance problems. Do NOT nitpick style, naming conventions, minor formatting, or subjective preferences — only flag issues that could break functionality or cause real harm in production. |
|
Claude finished @axisrow's task in 1m 19s —— View job PR #179 Review
Two critical issues found in 1. Shell injection via unquoted
|
Addresses Claude review findings on #179: 1. The previous version interpolated token/login directly into the eval block. Yandex OAuth tokens use base64url alphabet and logins are constrained, so a real-world injection is unlikely — but 1Password and Bitwarden references could return arbitrary strings, and a token containing a shell metacharacter would either truncate the value or execute arbitrary code. Wrap both values with shlex.quote(). 2. get_credentials() returns Tuple[str, Optional[str]] — login can be None for profiles created without a login. The shell guard above only checks `has_token=yes`, so a token-without-login profile would reach the eval and export the literal string "None". Add the matching `if not token or not login` check inside the Python helper and propagate its non-zero exit through a `if ! resolved=$(...)` wrapper (eval swallows subshell exit codes otherwise). No behaviour change for callers with a fully-populated profile or env-supplied credentials. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
@claude review. Focus on critical issues: bugs, security vulnerabilities, logical errors, data loss risks, performance problems. Do NOT nitpick style, naming conventions, minor formatting, or subjective preferences — only flag issues that could break functionality or cause real harm in production. |
|
Claude finished @axisrow's task in 1m 32s —— View job PR #179 Review
Both critical issues flagged in the previous review cycle are resolved in commit Previous findings — resolvedFinding 1 — Shell injection via unquoted Finding 2 — if not token or not login:
sys.exit("ERROR: auth profile resolved a token but no login. Re-run \"direct auth login\".")The
|
Click 8.x changed the missing-option error format from "No such option: --foo" to "No such option '--foo'. Did you mean '--bar'?" This broke three legacy-flag rejection tests on Python 3.11/3.13 CI runners (3.9 still ships an older Click). Split each assertion into two checks — "No such option" + the flag name — which match both the old and new formats. Fixes the CI failure inherited from main on #177. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…177) * test(v4_live): add live contracts for v4forecast, v4tags, v4wordstat Refs #174. - wordstat lifecycle (create/list/delete) under v4_live_read marker - forecast lifecycle (create/list/delete) under v4_live_read marker - tags GetCampaignsTags read-only via existing _campaign_id helper - tags GetBannersTags read-only via existing _campaign_id helper GetForecast and GetWordstatReport are out of scope (require polling). UpdateBannersTags / UpdateCampaignsTags are out of scope (write). Tests auto-skip without YANDEX_DIRECT_TOKEN / YANDEX_DIRECT_LOGIN. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v4_live): gate wordstat/forecast lifecycle behind LIVE_WRITE opt-in Addresses Codex review on #177: CreateNew*/Delete* are write operations and must not run as part of the read-only v4_live_read tier. Follows the existing pattern of test_v4_live_create_invoice_contract_opt_in_write (line 140) — env-gate inside the test body with explicit _opt_in_write suffix in the function name. Behaviour: - Without YANDEX_DIRECT_LIVE_WRITE=1: tests skip (even with token+login). - With YANDEX_DIRECT_LIVE_WRITE=1: tests run and mutate the live account with a single create/delete cycle. Tags read-only tests (GetCampaignsTags, GetBannersTags) are unchanged — they remain pure reads under v4_live_read. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test: relax 'No such option' asserts for Click 8.x format Click 8.x changed the missing-option error format from "No such option: --foo" to "No such option '--foo'. Did you mean '--bar'?" This broke three legacy-flag rejection tests on Python 3.11/3.13 CI runners (3.9 still ships an older Click). Split each assertion into two checks — "No such option" + the flag name — which match both the old and new formats. Fixes the CI failure inherited from main on #177. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v4_live): apply Claude review polish (next/single-arg/TODOs) Addresses Claude review on #177: - Use next() to fetch the just-created report/forecast from the list and validate its schema directly, instead of asserting on reports[0] which may belong to a prior run. - Drop redundant third argument to call_v4(..., "List*", None) — param defaults to None. - Add TODOs flagging two assertions to tighten after first live run: Delete* return value (== 1 per docs) and GetBannersTags field set (likely {BannerID, TagIDS}, mirroring UpdateBannersTags writes). Behaviour unchanged for both skip path and live path; lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v4_live): resolve credentials from direct auth profile when env is unset Companion fix to #179 (auth-profile fallback in WRITE_SANDBOX runners). Previously _credentials() silently skipped all v4_live_read tests for profile-only users — manual pytest -m v4_live_read showed no signal. Now: when env vars are missing, try direct_cli.auth.get_credentials() to resolve the profile (matches the priority used by every CLI subcommand). Skip message also tells the user both options. Refs #178. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v4_live): move write lifecycle probes to integration_live_write via CLI Addresses Codex review finding #2 on #179: write-mutating tests were inheriting the v4_live_read file-level marker, breaking the trust boundary between read-only and write test suites. - Removes test_v4_live_wordstat_lifecycle_contract_opt_in_write and test_v4_live_forecast_lifecycle_contract_opt_in_write from tests/test_v4_live_contracts.py (which is correctly marked v4_live_read). - Adds test_live_draft_v4wordstat_lifecycle and test_live_draft_v4forecast_lifecycle to tests/test_integration_live_write.py under the integration_live_write marker. - Rewrites them to use the CLI (via _invoke_live / CliRunner) instead of native call_v4, matching the convention of every other live-write test in that file. - Drops in-test YANDEX_DIRECT_LIVE_WRITE checks — the file-level skipif already enforces the env gate. Verified: pytest --collect-only -m v4_live_read tests/test_v4_live_contracts.py → 9 read-only tests (no lifecycle probes). pytest --collect-only -m integration_live_write tests/test_integration_live_write.py → both v4 lifecycle tests included. env -u YANDEX_DIRECT_LIVE_WRITE pytest <…lifecycle…> → SKIPPED via file-level skipif. Codex review finding #1 (bare `pytest` hits live API when profile is active) is intentionally deferred: project decision is that v4 live read tests should run whenever a profile is available. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v4_live): record VCR cassettes for v4wordstat/v4forecast lifecycle Adds @pytest.mark.vcr to the two new v4 lifecycle tests and commits their recorded cassettes so `pytest -m integration_live_write` no longer hits the production API on every run, matching the contract documented in README and pyproject.toml. Also extends `_before_record_request` in tests/conftest.py to redact the OAuth token from the request body: the v4 JSON API embeds `"token":"..."` inside the payload (not just the Authorization header), so the existing header-only filter would have leaked it into the recorded YAML. Addresses Codex adversarial review feedback on #177. * test(v4_live): move v4 report lifecycle to opt-in tier, add orphan store Codex's second adversarial review flagged two real issues confirmed by live verification: 1. tests/test_v4_live_contracts.py:_credentials() falls back to the active `direct auth` profile when env vars are missing. This is intentional but was never documented, and contradicts CLI's own priority chain (where the profile wins over env). Document the inverted, tests-only contract in _credentials() docstring, CLAUDE.md and README.md (EN+RU). No code change — behaviour is already correct (env wins; profile is fallback; otherwise skip). 2. test_live_draft_v4{wordstat,forecast}_lifecycle in tests/test_integration_live_write.py created account-level v4 reports under a marker that promises "only disposable draft resources". Move the two tests to tests/test_v4_live_contracts.py under the existing `_opt_in_write` pattern, gated by YANDEX_DIRECT_V4_LIVE_REPORT_WRITE=1. Delete the two corresponding cassettes (no replay for this tier — opt-in live only). Rename the remaining 18 v5 tests and cassettes from `test_live_draft_*` to `test_v5_live_draft_*` so v5 vs v4 is immediately visible at the test name level. Add tests/_orphan_store.py: a small atomic JSON store at ~/.direct-cli/test-orphans.json that records created v4 report IDs. Each opt-in test calls drain() on entry to retry deletions left over from an interrupted previous run, add() right after create, and remove() after a successful delete. If delete fails (network drop, SIGKILL), the ID stays in the store and gets cleaned up next time. Atomic write pattern modeled on direct_cli/auth.py:144-155. Verified: - pytest -m integration_live_write replay: 10 passed + 7 skipped (the same 18-test surface, minus the 2 removed v4 tests; cassettes resolve via new v5_ names). - pytest tests/test_v4_live_contracts.py without env: both new tests skip as expected. - orphan store: simulated network failure scenario — ID persists after failed delete, gets picked up by drain() on next run, store empties. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(v5): rename test_integration_live_write to test_v5_live_write Align the file name with the test names. After the previous commit renamed the 18 functions to `test_v5_live_draft_*`, the file itself still said `test_integration_live_write.py` — function prefix and file name now point at different things ("v5" vs. nothing). Renaming makes the file/test/cassette directory all say v5. The pytest marker `integration_live_write` is kept untouched — it describes the run contract (`-m integration_live_write`), not the file name, and renaming it would break the CI/dev invocation surface. git mv: - tests/test_integration_live_write.py → tests/test_v5_live_write.py - tests/cassettes/test_integration_live_write/ → tests/cassettes/test_v5_live_write/ (pytest-recording derives the cassette directory from the module name, so the directory has to track the file). Doc references updated: - tests/API_ISSUE_AUDIT.md - tests/API_COVERAGE.md Verified: 18 tests collect under the new path, replay returns the same 10 passed + 7 skipped as before, no leftover refs to the old name in the repo (only in `direct_cli.egg-info/SOURCES.txt`, which is a build artifact and not committed). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test: broaden exception handlers in test bootstrap and orphan store Address two real defects flagged in PR #177 review on commit a200daa. 1. tests/test_v4_live_contracts.py::_credentials() previously caught only (ValueError, RuntimeError, ImportError) from direct_cli.auth.get_credentials(None, None). Tracing the call path showed two reachable leak points after a successful OAuth refresh: - OSError from save_auth_store → _write_json (os.chmod + mkstemp on ~/.direct-cli/ — unconditional, no swallow); - json.JSONDecodeError from parsing a malformed refresh-token response body (auth.py:534). Either would turn a "skip when creds unavailable" bootstrap into a hard test error on a dev machine with a stale profile but no env vars. Widen the except to include OSError and json.JSONDecodeError, and add a comment so the next reader sees why these two specifically. The other failure modes Copilot mentioned (HTTPError/URLError, KeyError/TypeError) are already trapped inside auth.py and cannot escape — confirmed by reading load_auth_store, get_oauth_profile, refresh_access_token, and validate_oauth_profile. 2. tests/_orphan_store.py::_read() had its list comprehension `[int(x) for x in v if isinstance(x, (int, str))]` outside the try/except, so a corrupted bucket like {"v4wordstat": ["abc"]} would raise ValueError up through add/remove/drain — contradicting the module docstring promise that "all read/write errors are swallowed silently so tests are never broken by store corruption." Move the comprehension inside the try and extend except with (ValueError, TypeError). Also drop redundant FileNotFoundError (subclass of OSError). Verified with hand-corrupted JSON: "abc" yields {}, [null, 456] yields {"v4wordstat": [456]}, plain garbage yields {}. Codex's P2 (skip v4 tests entirely without explicit env credentials) is left unaddressed: the env > profile > skip contract is the documented and intended behavior — see CLAUDE.md, README.md (EN+RU), and the _credentials() docstring. Reply posted on the thread. Copilot's other comments triaged: - "PR description doesn't reflect actual scope": process feedback, not a code issue. - "Weak BannerID assert": intentional per the file's "confirm at first live run" pattern (matches sibling v4 read-only tests). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: axisrow <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Summary
WRITE_SANDBOX runners now fall back to the
direct auth loginprofile whenYANDEX_DIRECT_TOKEN/YANDEX_DIRECT_LOGINaren't in env. Matches the existing behaviour ofscripts/test_safe_commands.sh.Closes #178.
Problem
Same population can't run v4 live tests from #177 — they silently skip. The companion fix for
tests/test_v4_live_contracts.py::_credentials()will go into #177 directly to avoid cross-PR conflict.Changes
scripts/test_sandbox_write.shdirect auth statusfor the literalhas_token=yesmarker (the command always exits 0, so a non-zero exit can't be the gate).direct_cli.auth.get_credentials(None, None)andeval-exports them so the Python child and itsdirect --sandbox …subprocesses inherit them.scripts/sandbox_write_live.py:validate_environment()get_credentials()for direct Python invocations that bypass the shell wrapper.os.environso spawneddirect --sandbox …subprocesses see the credentials.CI is not affected — these runners aren't called from any
.github/workflows.Test plan
Verified locally on macOS with active OAuth profile and no env vars:
env -u YANDEX_DIRECT_TOKEN -u YANDEX_DIRECT_LOGIN bash scripts/test_sandbox_write.sh --help→ resolves through profile, Python runner exec'd ✓env HOME=/tmp/empty env -u … bash scripts/test_sandbox_write.sh→ cleanERROR: no credentials. Use 'direct auth login', …, exit 1 ✓flake8 scripts/sandbox_write_live.pycleanblack --check scripts/sandbox_write_live.pycleanbash -n scripts/test_sandbox_write.shclean🤖 Generated with Claude Code