DownloadManager: add redact_url= kwarg to hide auth-bearing URLs in logs#136
Merged
ThomasFarstrike merged 2 commits intoMay 16, 2026
Conversation
`DownloadManager.download_url` currently prints the full request URL three times per download (start, finished, exception path), plus the entire response headers dict. That's useful debug output for the typical callers fetching public URLs (app icons, OS updates, weather data) — but it silently leaks any auth secret embedded in a URL's path or query string. Two real cases discovered in the wild: 1. Lightning Piggy's on-chain wallet (LightningPiggyApp PR MicroPythonOS#25) queries `https://btc1.trezor.io/api/v2/xpub/zpub6q...?details=txs&tokens=derived` to fetch balance + transactions for a watch-only xpub. The xpub itself is in the path. A leaked xpub exposes the wallet's entire past + future address derivation tree to whoever reads the log — non-recoverable confidentiality damage even though funds custody is unaffected. 2. Any future caller using API-key-in-URL or OAuth-token-in-URL authentication. Pattern is common enough (LNBits, BlueWallet's LNDHub, some HTTPS RPC endpoints) that the leak surface is wider than just Lightning Piggy. The Lightning Piggy app already scrubs xpubs from its own print() calls, but DownloadManager's prints sit below that and re-leak the URL on every poll. This PR adds a `redact_url=False` kwarg (default preserves current behaviour). When set True: - The URL is logged as `scheme://host[:port]/...REDACTED...` instead of full. The host is intentionally kept so failure triage (DNS, connectivity, wrong endpoint) is still possible. - The response-headers dump is suppressed entirely (replaced with `<redacted>`). Response headers often contain `set-cookie`, `cf-ray`, and other correlatable session tokens. - Exception messages have the URL substring scrubbed before printing (aiohttp's ClientConnectorError typically embeds the URL). Default False is deliberate — most callers fetch public URLs and want the full log line for diagnostics. Callers opt in: await DownloadManager.download_url(url, redact_url=True) Tests (`tests/test_download_manager.py`): - 6 unit tests for the new `_safe_url` helper (path-with-query strip, port handling, naked-host pass-through, malformed-URL placeholder, belt-and-braces "secret substrings never appear in output" guard). - 2 mock-surface tests confirming the kwarg flows through correctly and defaults to False. - All 22 existing tests continue to pass (default behaviour unchanged). $ bash tests/unittest.sh tests/test_download_manager.py Ran 30 tests OK The Lightning Piggy on-chain wallet PR will follow up on its side to pass `redact_url=True` once this lands; that's a one-line change in `onchain_wallet.py:fetch_balance_and_payments()`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3 tasks
Contributor
Author
CONTRIBUTING.md merge-checklist auditWalking through each item explicitly so the maintainer doesn't have to re-derive it:
AGENTS.md style:
Companion docs PR: MicroPythonOS/docs#6 — ready to merge in lockstep with this one. |
This was referenced May 13, 2026
PR MicroPythonOS#136 added URL scrubbing to the `except Exception as e:` handler in `_download_url_async`, but the verification of that branch was deferred to a hardware smoke-test. Move it under unit-test coverage: - Install a fake `aiohttp` into `sys.modules` so the function-local `import aiohttp` resolves to a stub whose `ClientSession.get()` raises a `RuntimeError(message)` where `message` contains the full request URL (mimicking aiohttp's `ClientConnectorError` behaviour without needing the network). - Capture `print(...)` via builtins monkey-patch, then assert: - `redact_url=True` strips the secret-bearing URL substring from the printed exception line and substitutes the `scheme://host/...REDACTED...` form - `redact_url=False` preserves the full URL in the printed line (regression guard for the default behaviour) - Use a plain class instance for the fake module — MicroPython's unittest port doesn't support `type(sys)(...)` to mint a new module object, but `sys.modules[name]` accepts any object. `bash tests/unittest.sh tests/test_download_manager.py` → 32 tests OK (was 30). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DownloadManager.download_urlprints the full request URL three times per download (start, finished, exception path) plus the entire response-headers dict. Great for the typical callers fetching public URLs (app icons, OS updates, weather), but it silently leaks any auth secret embedded in the URL's path or query string.Two real cases motivating this PR:
Lightning Piggy's on-chain wallet (LightningPiggyApp PR #25) queries
https://btc1.trezor.io/api/v2/xpub/zpub6q…?details=txs&tokens=derivedto fetch balance + transactions for a watch-only xpub. The xpub itself sits in the URL path. A leaked xpub exposes the wallet's entire past + future address derivation tree to whoever reads the log — non-recoverable confidentiality damage even though funds custody is unaffected. The app already scrubs xpubs from its ownprint()calls, butDownloadManager's prints sit below that and re-leak on every poll.Any future caller using API-key-in-URL or token-in-URL auth. Common pattern (LNBits, BlueWallet's LNDHub, some HTTPS RPC endpoints) — the leak surface is wider than just Lightning Piggy.
Behaviour
Adds a
redact_url=Falsekwarg (default preserves current behaviour). WhenTrue:scheme://host[:port]/...REDACTED...instead of full. The host is intentionally kept so failure triage (DNS / connectivity / wrong endpoint) is still possible.<redacted>). Response headers often carryset-cookie,cf-ray, and other correlatable session tokens.ClientConnectorErrortypically embeds the URL).Default
Falseis deliberate — most callers fetch public URLs and want the full log line for diagnostics. Callers explicitly opt in:Tests
tests/test_download_manager.py— added 3 new test classes (10 tests):TestSafeUrl(6 tests) for the new_safe_urlhelper — path-with-query strip, port handling, naked-host pass-through, malformed-URL placeholder, belt-and-braces "secret substrings never appear in output" guard.TestRedactUrlKwarg(2 tests) confirming the kwarg flows through the call surface (sync wrapper + async impl + mock) and defaults toFalse.TestRedactedExceptionPath(2 tests, commit64963f9) — end-to-end coverage of the URL-bearing exception scrubbing path. Installs a fakeaiohttpintosys.modulesso the function-localimport aiohttpresolves to a stub whoseClientSession.get()raisesRuntimeError(message)with the full URL in the message (mimics aiohttp'sClientConnectorErrorwithout needing the network), then capturesprint(...)and asserts the secret-bearing substring is scrubbed underredact_url=Trueand preserved underredact_url=False.All 22 existing tests continue to pass (default behaviour unchanged):
Follow-up
Lightning Piggy's on-chain wallet (PR #25) will follow up on its side to pass
redact_url=Trueonce this lands. That's a one-line change inonchain_wallet.py:fetch_balance_and_payments().Test plan
bash tests/unittest.sh tests/test_download_manager.py)64963f9): fake-aiohttp injection deterministically exercises theexcept Exception as e: ... url in err_strbranch and verifies both the scrubbed (redact_url=True) and preserved (redact_url=False) print outputredact_url=Trueshowshttps://btc1.trezor.io/...REDACTED...in serial logs. Gated on LightningPiggy #25 follow-up adding the kwarg — will tick once that lands and the integrated build is exercised on device🤖 Generated with Claude Code