Skip to content

DownloadManager: add redact_url= kwarg to hide auth-bearing URLs in logs#136

Merged
ThomasFarstrike merged 2 commits into
MicroPythonOS:mainfrom
bitcoin3us:security/download-manager-redact-url
May 16, 2026
Merged

DownloadManager: add redact_url= kwarg to hide auth-bearing URLs in logs#136
ThomasFarstrike merged 2 commits into
MicroPythonOS:mainfrom
bitcoin3us:security/download-manager-redact-url

Conversation

@bitcoin3us
Copy link
Copy Markdown
Contributor

@bitcoin3us bitcoin3us commented May 13, 2026

Summary

DownloadManager.download_url prints the full request URL three times per download (start, finished, exception path) plus the entire response-headers dict. Great for the typical callers fetching public URLs (app icons, OS updates, weather), but it silently leaks any auth secret embedded in the URL's path or query string.

Two real cases motivating this PR:

  1. Lightning Piggy's on-chain wallet (LightningPiggyApp PR #25) queries
    https://btc1.trezor.io/api/v2/xpub/zpub6q…?details=txs&tokens=derived to fetch balance + transactions for a watch-only xpub. The xpub itself sits in the URL path. A leaked xpub exposes the wallet's entire past + future address derivation tree to whoever reads the log — non-recoverable confidentiality damage even though funds custody is unaffected. The app already scrubs xpubs from its own print() calls, but DownloadManager's prints sit below that and re-leak on every poll.

  2. Any future caller using API-key-in-URL or token-in-URL auth. Common pattern (LNBits, BlueWallet's LNDHub, some HTTPS RPC endpoints) — the leak surface is wider than just Lightning Piggy.

Behaviour

Adds a redact_url=False kwarg (default preserves current behaviour). When True:

  • URL logged as scheme://host[:port]/...REDACTED... instead of full. The host is intentionally kept so failure triage (DNS / connectivity / wrong endpoint) is still possible.
  • Response-headers dump is suppressed entirely (replaced with <redacted>). Response headers often carry set-cookie, cf-ray, and other correlatable session tokens.
  • Exception messages have the URL substring scrubbed before printing (aiohttp's ClientConnectorError typically embeds the URL).

Default False is deliberate — most callers fetch public URLs and want the full log line for diagnostics. Callers explicitly opt in:

await DownloadManager.download_url(url, redact_url=True)

Tests

tests/test_download_manager.py — added 3 new test classes (10 tests):

  • TestSafeUrl (6 tests) for the new _safe_url helper — path-with-query strip, port handling, naked-host pass-through, malformed-URL placeholder, belt-and-braces "secret substrings never appear in output" guard.
  • TestRedactUrlKwarg (2 tests) confirming the kwarg flows through the call surface (sync wrapper + async impl + mock) and defaults to False.
  • TestRedactedExceptionPath (2 tests, commit 64963f9) — end-to-end coverage of the URL-bearing exception scrubbing path. Installs a fake aiohttp into sys.modules so the function-local import aiohttp resolves to a stub whose ClientSession.get() raises RuntimeError(message) with the full URL in the message (mimics aiohttp's ClientConnectorError without needing the network), then captures print(...) and asserts the secret-bearing substring is scrubbed under redact_url=True and preserved under redact_url=False.

All 22 existing tests continue to pass (default behaviour unchanged):

$ bash tests/unittest.sh tests/test_download_manager.py
Ran 32 tests
OK

Follow-up

Lightning Piggy's on-chain wallet (PR #25) will follow up on its side to pass redact_url=True once this lands. That's a one-line change in onchain_wallet.py:fetch_balance_and_payments().

Test plan

  • All 32 tests pass on macOS desktop (bash tests/unittest.sh tests/test_download_manager.py)
  • Existing tests (default behaviour) unchanged
  • URL-bearing exception path covered by unit tests (commit 64963f9): fake-aiohttp injection deterministically exercises the except Exception as e: ... url in err_str branch and verifies both the scrubbed (redact_url=True) and preserved (redact_url=False) print output
  • Hardware end-to-end smoke test: an LP onchain wallet call with redact_url=True shows https://btc1.trezor.io/...REDACTED... in serial logs. Gated on LightningPiggy #25 follow-up adding the kwarg — will tick once that lands and the integrated build is exercised on device

🤖 Generated with Claude Code

`DownloadManager.download_url` currently prints the full request URL three
times per download (start, finished, exception path), plus the entire
response headers dict. That's useful debug output for the typical callers
fetching public URLs (app icons, OS updates, weather data) — but it
silently leaks any auth secret embedded in a URL's path or query string.

Two real cases discovered in the wild:

  1. Lightning Piggy's on-chain wallet (LightningPiggyApp PR MicroPythonOS#25) queries
     `https://btc1.trezor.io/api/v2/xpub/zpub6q...?details=txs&tokens=derived`
     to fetch balance + transactions for a watch-only xpub. The xpub
     itself is in the path. A leaked xpub exposes the wallet's entire
     past + future address derivation tree to whoever reads the log —
     non-recoverable confidentiality damage even though funds custody
     is unaffected.

  2. Any future caller using API-key-in-URL or OAuth-token-in-URL
     authentication. Pattern is common enough (LNBits, BlueWallet's
     LNDHub, some HTTPS RPC endpoints) that the leak surface is wider
     than just Lightning Piggy.

The Lightning Piggy app already scrubs xpubs from its own print() calls,
but DownloadManager's prints sit below that and re-leak the URL on
every poll.

This PR adds a `redact_url=False` kwarg (default preserves current
behaviour). When set True:

- The URL is logged as `scheme://host[:port]/...REDACTED...` instead of
  full. The host is intentionally kept so failure triage (DNS,
  connectivity, wrong endpoint) is still possible.
- The response-headers dump is suppressed entirely (replaced with
  `<redacted>`). Response headers often contain `set-cookie`, `cf-ray`,
  and other correlatable session tokens.
- Exception messages have the URL substring scrubbed before printing
  (aiohttp's ClientConnectorError typically embeds the URL).

Default False is deliberate — most callers fetch public URLs and want
the full log line for diagnostics. Callers opt in:

    await DownloadManager.download_url(url, redact_url=True)

Tests (`tests/test_download_manager.py`):
- 6 unit tests for the new `_safe_url` helper (path-with-query strip,
  port handling, naked-host pass-through, malformed-URL placeholder,
  belt-and-braces "secret substrings never appear in output" guard).
- 2 mock-surface tests confirming the kwarg flows through correctly
  and defaults to False.
- All 22 existing tests continue to pass (default behaviour unchanged).

  $ bash tests/unittest.sh tests/test_download_manager.py
  Ran 30 tests
  OK

The Lightning Piggy on-chain wallet PR will follow up on its side to
pass `redact_url=True` once this lands; that's a one-line change in
`onchain_wallet.py:fetch_balance_and_payments()`.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@bitcoin3us
Copy link
Copy Markdown
Contributor Author

CONTRIBUTING.md merge-checklist audit

Walking through each item explicitly so the maintainer doesn't have to re-derive it:

Item Status Detail
1. Expand CHANGELOG.md "Future release" Bullet added under Frameworks
2. Increment app version in META-INF/MANIFEST.JSON N/A Framework change, not an app
3. Update documentation repository MicroPythonOS/docs#6 — adds the new kwarg to the parameters table + a "Redacting Sensitive URLs" section covering when to opt in and what gets redacted
4. Revise MAINTAINERS.md N/A No new boards
5. Settings migration N/A No settings touched
6. Unit tests 8 new tests in tests/test_download_manager.py covering the _safe_url helper and the kwarg's flow through both sync wrapper and async impl. All 30 download_manager tests pass
7. Separate non-functional changes Single concern, no drive-by formatting

AGENTS.md style:

Rule Status
ruff double-quotes
Test placement (mpos.testing mocks where applicable) ✅ Uses existing MockDownloadManager from mpos.testing.mocks
Test naming convention (test_*.py)

Companion docs PR: MicroPythonOS/docs#6 — ready to merge in lockstep with this one.

PR MicroPythonOS#136 added URL scrubbing to the `except Exception as e:` handler in
`_download_url_async`, but the verification of that branch was deferred
to a hardware smoke-test. Move it under unit-test coverage:

- Install a fake `aiohttp` into `sys.modules` so the function-local
  `import aiohttp` resolves to a stub whose `ClientSession.get()`
  raises a `RuntimeError(message)` where `message` contains the full
  request URL (mimicking aiohttp's `ClientConnectorError` behaviour
  without needing the network).
- Capture `print(...)` via builtins monkey-patch, then assert:
  - `redact_url=True` strips the secret-bearing URL substring from
    the printed exception line and substitutes the
    `scheme://host/...REDACTED...` form
  - `redact_url=False` preserves the full URL in the printed line
    (regression guard for the default behaviour)
- Use a plain class instance for the fake module — MicroPython's
  unittest port doesn't support `type(sys)(...)` to mint a new
  module object, but `sys.modules[name]` accepts any object.

`bash tests/unittest.sh tests/test_download_manager.py` → 32 tests OK
(was 30).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ThomasFarstrike ThomasFarstrike merged commit 6a59342 into MicroPythonOS:main May 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants