Merge all keepkey/python-keepkey upstream branches into alpha - #109

Blind-sign policy gating (AdvancedMode blocking) is 7.15+ scope.
On 7.10-7.14 blind signing is always permitted. Remove
test_ethereum_blind_sign_blocked which expected 7.15 firmware
behavior and update report section V accordingly.

The test's own docstring says the feature it exercises requires firmware
7.15.1+, but the gate admitted 7.15.0. Any 7.15.0 build therefore runs a
test for behaviour that release is not expected to have: entering a word
outside the BIP-39 wordlist returns a CharacterRequest for the next word
rather than a Failure, and the assertion fails.

No behaviour change -- the gate now matches the docstring beside it.

keepkey#216 landed as one squashed commit whose content had already reached keepkey#197 by
another route, so most of the 31 conflicts are the two branches doing the same
thing in different words. Resolved by hand, hunk by hunk; no --ours/--theirs.
Every keepkey#216 addition is verified present in the merged tree, including the
files that never conflicted (test_msg_display_disclosure.py, the eip155
vectors, the ripple/ping/tron/eth-signtx disclosure tests, and the
`chain_id is not None` fix in keepkeylib/client.py -- that one matters, it is
what lets an explicit chain_id=0 reach the wire to be refused).

tests/test_sign_typed_data.py (1 hunk) -- keepkey#197
  Same policy call on both sides; the only difference is `with open(...)` vs
  open/close and one extra clause of comment. Nothing asserted differs.

tests/test_msg_thorchain_signtx.py (2 hunks) -- keepkey#197, and this one is a real
divergence, not a restatement.
  keepkey#197 repointed test_sign_eth_add_liquidity at the firmware-pinned THORChain
  router (to=0xd37bbe57...); that line auto-merged, so it IS the transaction
  the merged test signs. keepkey#216 regenerated exact r/s against the OLD
  to=0x41e55600.... `to` is an RLP field of the legacy sighash, so keepkey#216's
  vectors do not describe the transaction under test and would fail. keepkey#197's
  structural assertions win on those grounds -- NOT because structural is
  preferable. They are genuinely weaker than what keepkey#216 had, and the superseded
  vectors are recorded in a comment at the assertion so the gap is visible and
  regenerable on-device. Same treatment for the memo-length hunk: both sides
  independently corrected 0x3b -> 0x3a (the memo is 58 bytes; the 59th was ABI
  padding), so the comments were combined rather than one dropped.

tests/test_msg_mayachain_signtx.py (2 hunks) -- keepkey#197
  Same router story (to=0xe3985e6b..., Maya router v4), so keepkey#216's exact r/s --
  byte-identical to the thorchain pair, same base fixture -- are stale here too.
  keepkey#197 does not merely relax: it reconstructs the sighash host-side and recovers
  the signer, so a wrong digest, wrong calldata or wrong key fails. Taking keepkey#216
  would also have re-added a stray `)` and restored @unittest.skip on
  test_mayachain_remove_liquidity, which keepkey#197 unskipped. Raw assertion count in
  this file drops 26 -> 13 because seven frozen-vector memo cases became one
  loop over the same seven memos and three skipped tests now actually run; memo
  coverage is unchanged at nine cases and none of them are skipped any more.

tests/test_msg_ton_signtx.py (9 hunks) -- union
  Both sides added the identical AdvancedMode opt-in to nine tests; keepkey#216 also
  carried the four-line reason. Kept keepkey#216's comment and call on all nine.
  keepkey#197 additionally gated three tests keepkey#216 did not -- the reject-path tests --
  and those auto-merged; annotated them, because opening the gate is what stops
  them passing for the wrong reason (with the gate shut the firmware refuses
  every TonSignTx and the raises-assertion is satisfied without the parser ever
  validating a field).

scripts/generate-test-report.py (17 hunks) -- keepkey#197
  Fifteen hunks are section-letter collisions: keepkey#216 filed Display Binding under
  'S' and Display Disclosure under 'D', but 'S' is already Solana and 'D' is
  already BIP-85 Child Derivation, so keepkey#216's own catalog carries each letter
  twice. keepkey#197 assigns 'J' and 'Q' and does not collide; _audit_catalog() asserts
  letter and id uniqueness and would have tripped on keepkey#216's scheme. Entries are
  otherwise identical -- same module, same test, same prose -- so nothing keepkey#216
  catalogued is lost, only relettered. Fixed one cross-reference keepkey#197's rename
  left dangling: J13's prose said "the control for S12", which now points at a
  Solana test; it means J12. The other two hunks are keepkey#197-only additions (the F,
  I, L, U, TD sections and MUST_RUN_MODULES) against an empty keepkey#216 side.

tests/test_msg_ethereum_erc20_0x_signtx.py -- union, and git got this one wrong
  Not one of the 31: it merged clean and merged badly. Both sides inserted an
  AdvancedMode opt-in into test__sign_transformERC20 at different offsets, so
  the merged file called apply_policy twice in a row. Deduped to one call
  carrying both rationales (keepkey#216's 1480-bytes-past-the-1024-chunk mechanism,
  which is what atlas J1 actually claims, plus keepkey#197's no-recognised-handler
  note). Folded in the comment keepkey#197 left three lines above it saying "No
  AdvancedMode policy is set here on purpose", which contradicted its own
  apply_policy call and both sides' findings; kept as an explicitly superseded
  note rather than deleted, so the behaviour change stays legible.

Verified: 0 conflict markers; every .py under tests/, scripts/ and keepkeylib/
parses; _audit_catalog() passes and the report renders 27 sections / 381
distinct tests; all 383 catalog rows resolve to a test that exists (V17-V67
against CLEARSIGN_FLOWS, which generates those methods by the same transform;
K3-K11 are native gtest suites the JUnit parser handles by bare classname).
Not verified: no test was executed against firmware. The one suite that runs
offline, test_storage_version_gate.py, needs the firmware sources checked out
above this tree and errors in setUp without them -- unchanged by this merge.

These three tests lost their emulator guard, and without it they do not
fail on the emulator -- they HANG. The ETH/FOX pool contract is not in the
token table, and approving an unknown token contract never completes, so
the device keeps answering while the approval never finishes. The socket
timeout never fires because reads keep succeeding, so the job goes silent
until CI kills it on its no-output timeout, taking every later test with
it. That is why the suite died at 34% having reported no failure.

Gated rather than deleted, so the coverage comes back on its own once the
firmware completes this path. Known-token approves still run here, and on
hardware the app exercises this path.

TestStorageVersionGateSource asserted STORAGE_VERSION_LAST_SHIPPED exists,
so all eleven of its tests failed at setUp against any firmware that
predates the storage version gate -- which is every branch except alpha.
That is what has been red on CircleCI, which builds firmware master.

The gate is a firmware FEATURE, so gate on the capability. A tree that
never had it has nothing here to assert.

The skip is deliberately narrow: if storage.c references the constant but
storage.h no longer defines it, that is the floor being deleted out from
under the static assert -- the exact regression this suite exists to catch
-- and it still FAILS. The skip cannot swallow the deletion it guards.

Verified against three real trees: 7.14.3 runs (floor 17), upstream develop
skips, alpha runs (V20, floor 17).

emulator-build-test duplicated the integration job in .github/workflows,
and did the same work worse in three ways:

  * it cloned firmware `master` unconditionally, so any branch carrying a
    newer test harness ran against older firmware and failed for reasons the
    PR did not cause. That is why it has been red since build #673 while
    GitHub Actions stayed green on the same commits.
  * it set no timeouts, so a hung test produced no output until CircleCI
    killed the job at ten minutes -- reported as a timeout rather than a
    failing test, and discarding every result after the hang.
  * it also ran the firmware's C++ firmware-unit suite, which covers firmware
    code that no change in this repo can affect, and which the firmware repo
    already runs in its own CI.

The integration job builds from current firmware and bounds every step with
timeout-minutes, so a hang fails fast and names the test.

master has no required status checks, so removing the context does not block
any pull request. Also corrects the ci.yml header, which still described
pulling a published DockerHub image -- the job stopped doing that when it
started building from firmware.

The bitcoin-only 7.14.3 release line carries the dice-entropy backport, and
no firmware between 7.14.3 and 7.15.0 exists without it, so the 7.15.0 gates
were exact only for the 7.15 line: against 7.14.3 the dice end-to-end test,
the ceremony re-entry regression (the host-chosen-seed guard), and report
Section K all skipped silently, and CI went green with zero device-level
dice coverage.

Gates move to 7.14.3, which is exact for the whole fleet. Section K retitled
accordingly.

…ability-gates

test(bitcoin-only): gate unsupported 7.15 handlers

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

…gn-tests

test: update 7.14.2 presign authentication ordering

The K section catalogued the four Dice.Mix* gtests by name; those went with
dice_mix(). The 7.15 bitcoin-only leg's catalog validation reported them
missing. Replaced with the eight gtests the unit actually has: the two
derivation vectors, the zero-draw vector, in-place aliasing, non-collision
with the old formula, exact-count, and the two bias-gate tests.

…lay name

Two assertions read the asset symbol out of the transaction the DEVICE
serialized and expected "HIVE". This file already knows better: _WIRE_SYMBOL
at the top records that the 2020 rebrand renamed the tokens but not their
on-chain serialization, and every operation this file builds itself is
assembled with "STEEM"/"SBD" (confirmed against
condenser_api.get_transaction_hex).

So these two assertions were pinning the firmware's own mistake: it wrote
"HIVE" where hived writes "STEEM", and the test agreed with it. The firmware
side is fixed on the 7.15 line; this makes the expectation match the chain.

kkemu_get_display() lit every nonzero shade while the DebugLink layout and the
capture ring used ordered dithering (display_mono_pixel_is_lit), so the dylib
transport this test reads produced a different frame than the device's other
evidence paths for the same screen -- and this golden pinned the odd one out.
The firmware side is aligned on the 7.15 line; this is the frame all three
serialisers now agree on.

…odes

The dice ceremony now opens an on-device selector (MIXED / DICE ONLY) and
both modes derive a seed the user can recompute. The old test expected the
roll screen first and a seed that mixed the host's EntropyAck bytes; it is
replaced by a shared driver plus three tests.

Expected values come from the published formulas, restated here and NOT read
back from the device: a fixture from the code under test would only prove the
firmware agrees with itself. The 24 device-entropy words MIXED shows are
decoded by a BIP-39 decoder written here, checksum included, so the words a
user would copy down are checked by code the device did not write and the
test does not depend on the mnemonic library version in the test image.

- mixed: 99 rolls, 24 words; seed = SHA256d(tag || device || SHA256(tag ||
  rolls)) from the shown words and the injected rolls, with the host's
  nonzero EntropyAck nowhere in it.
- only: 50 rolls, 12 words; seed = SHA256(rolls), host bytes ignored.
- biased: fifty ones is refused with SyntaxError before any digest.

The roll pattern's top-up chunk is now uniform so it clears the 30% bias
gate at both targets (max face 17/99, 9/50).

This lives on a feature branch, not canonical reconcile/upstream-sync: the
release products do not carry the selector yet, and a version gate cannot
tell the 7.15 candidate from the 7.15 unit that does. Canonical takes it when
both products do.

…e-protocol

The checked-in Solana bindings predate the pinned device-protocol: they lack
SolanaSignTx.clearsign_certificate (field 13), which up/release-protocol has
carried since f54f0a7. Regenerated inside kktech/firmware:v8, the pinned
generator, so the old-style _pb2 stays loadable by the test image's runtime.
Surfaced by regenerating for the dice_only field; committed separately
because it is not part of that change.

… gating

The dice mode is now a host-side selection made before the ceremony starts,
so a wallet can explain what is coming: ResetDevice.dice_entropy alone is
MIXED, with dice_only it is DICE ONLY. The device answers with a consent
screen naming the mode it was asked for; holding proceeds, and the only "no"
is cancelling the reset. The on-device selector from the previous revision is
gone, and with it the DebugLink '1'/'2' injection the tests used.

Bindings regenerated from device-protocol feat/dice-only-field @ 451e9a7
(canonical up/release-protocol + the one field) inside kktech/firmware:v8, the
pinned generator. The regenerated diff is the new field and the shifted
descriptor offsets only.

Expected values still come from the published formulas restated here, not
from the device, and the 24 device words are decoded by a checksum-verified
BIP-39 decoder written here. Adds a test that dice_only without dice_entropy
is refused with SyntaxError before any screen.

… the driver

Findings from an adversarial review of the unit, all confirmed against the
firmware's own font tables and message pump:

- The full 32-byte digest wraps to two constant-power subpages, and under
  DEBUG_LINK each subpage after a debug decision raises its own ButtonRequest.
  The driver acked once and asserted EntropyRequest, so both verifiability
  tests would have failed before proving anything. It now holds through every
  digest page, as the backup-word loop already did.
- _inject_rolls diverged from dice_input_collect() once the target was reached:
  the device stops consuming a chunk at that point, undo included, and leaves
  the roll screen, so a later chunk would arrive at the digest confirm as a
  "no" decision. The host simulation now mirrors that exactly and stops
  sending.
- The dice tests were gated on a version. Firmware without the unit skips the
  unknown dice_only field and runs the older ceremony, so a version gate fails
  red on such a build -- and a host on the same signal would derive a
  different wallet without complaint. A Features.supports_dice_modes
  capability now gates both (requires_dice_modes(), after requires_taproot).

Adds tests for the two refusals the review asked for -- dice with no_backup,
and Cancel at the consent screen leaving nothing armed -- and catalogues all
six dice tests in the report atlas in place of the renamed original, with
screen lists matching what each captures.

Bindings regenerated inside kktech/firmware:v8 from device-protocol
feat/dice-only-field @ fbaf8ec (adds Features.supports_dice_modes = 28).

The K section catalogued the four Dice.Mix* gtests by name; those went with
dice_mix(). The 7.15 bitcoin-only leg's catalog validation reported them
missing. Replaced with the eight gtests the unit actually has: the two
derivation vectors, the zero-draw vector, in-place aliasing, non-collision
with the old formula, exact-count, and the two bias-gate tests.

ci: test Python harness against current firmware audit heads

test: two expectations that pinned firmware defects (Hive wire symbol, 1-bit frame hash)

Brings keepkey/python-keepkey's consolidated harness line (PR keepkey#197 to
upstream master; contains master, hive wire-symbol fix, 7.15 audit fixes,
bitcoin-only capability gates) into fork alpha: 84 commits.

Resolution rules:
- device-protocol pin, .gitmodules and generated bindings: alpha. Upstream's
  dp 27d3fa1 lacks the ERC-7730 and dice-mode protocol (moving to it deletes
  ~590 proto lines); bindings stay canonical-protoc (py3.6/protobuf 3.17).
- Harness (conftest network guard, common, client capture): upstream. Its
  guard admits only the exact configured UDP emulator endpoints plus owned
  leases. Removed the duplicate screenshot-selection hook alpha had ported.
- generate-test-report.py: upstream, keeping --build-variant as an alias of
  --variant (firmware CI passes --build-variant), alpha's --section option,
  and alpha's uniswap-liquidity must-run floor (GH #516) + its bitcoin-only
  waiver.
- test_msg_resetdevice.py: dice-modes version (f00e62f); firmware alpha
  runs the consented MIXED/ONLY flow, reconcile still has the old one.
- test_storage_version_gate.py: alpha's framed-record before/after pair
  (refresh_crc + sector) — firmware alpha stores framed journal records.
- test_msg_solana_signtx.py: alpha's stake layouts (validated against
  firmware alpha in CI) and the plain-text SignMessage test.
- test_msg_eos_signtx.py: alpha's version-gated golden.
- Dropped the unused noncanonical taproot prevtx fixture (upstream fc12c6d).
- CI configs: alpha's firmware targets; upstream's SHA check and the
  printf fix for the backtick-in-echo summary bug, applied to all four.

Offline: test_tx_fixture_integrity, test_network_policy, protocol bindings
15 passed; tx_fixture_manifest --check clean.

The host-selected dice modes with on-device consent that firmware alpha
implements (MIXED, ONLY, bias gate, supports_dice_modes gating) plus the
report catalog entries for the new native dice tests. Bindings and the
device-protocol pin stay alpha's (canonical protoc; already carry the
dice fields). Dropped the requires_dice_modes copy alpha had ported.

Content was already on alpha under different commits; this records the
ancestry. Kept alpha's KK_EXPECTED_FIRMWARE_REVISION provenance check
(upstream's GITHUB_SHA is the synthetic PR merge ref) and alpha's
device-protocol pin.

Takes the CircleCI retirement (alpha's workflow already documents it as
retired). Keeps alpha's integration workflow, which runs against the fork
firmware candidates; this branch's version runs against a published
emulator image and so tests stale firmware by its own note.

Ancestry only, no tree change:
- feat/dice-modes: all 5 patches present via feat/dice-modes-rebased
- fix/defer-blind-sign-to-715: patch present
- feat/test-report-generator: the original March generator; superseded by
  the current scripts/generate-test-report.py
- pre-squash-197-backup: 3428e0b (storage-gate skip) already present via
  reconcile; 1f6c240 (skip the FOX-pool uniswap tests on the emulator)
  intentionally NOT taken -- those tests no longer hang, they assert the
  unlimited-approve refusal and are must-run (GH #516)

The debug build raises one ButtonRequest per physical subpage and every
subpage of a word group reports the same reset_word; whether a group
spills to a second subpage depends on glyph widths. With a random
mnemonic this made the test flaky ("Invalid mnemonic, are words in
correct order?"): it passed and then failed on consecutive CI runs of
the same firmware. Same de-duplication test_msg_resetdevice already uses.

…ctations

8d3f4cc aligned these to keepkey/release/7.15 f3d9e058e, which is wrong
on the wire in both places:

- XRP: MemoData is Blob field 13 (0x7D). 0x72 is MessageKey, which
  rippled rejects inside a Memo object (ripple-binary-codec
  definitions.json). Every other firmware line, alpha included, emits 0x7D.
- Hive: memo_key is optional<> in account_update2 (op 43) in both dhive
  and hive-tx; it is mandatory only in legacy account_update (op 10).
  The builder dropped the presence byte, so firmware read 0x02 as a bool
  and failed with "malformed". Vectors 2/3 now carry a present memo key
  and must be refused as an authority change on every line.