Skip to content

Mob.Audio output probes: verify sound is actually working#54

Merged
GenericJam merged 3 commits into
masterfrom
feat/audio-output-probes
Jul 7, 2026
Merged

Mob.Audio output probes: verify sound is actually working#54
GenericJam merged 3 commits into
masterfrom
feat/audio-output-probes

Conversation

@GenericJam

@GenericJam GenericJam commented Jun 29, 2026

Copy link
Copy Markdown
Owner

What

The audio analog of screenshot/3: read-only probes that answer "is sound actually coming out right now."

  • Mob.Audio.output_status/0%{volume, muted, route, other_audio}. Cheap, no permission. Catches the common silence causes (muted, volume 0, dead route). iOS AVAudioSession / Android AudioManager.
  • Mob.Audio.output_level/1{rms_db, peak_db} | :silent | {:error, reason}. Reads actual signal energy — the part output_status and adb can't answer. source: :mob (default) meters Mob.Audio's own player (iOS AVAudioPlayer metering; Android Visualizer on the player's own session, needs runtime RECORD_AUDIO). source: :mix returns {:error, :unsupported_on_platform}.

Scope correction from device verification

The original design had :mix tap the global output mix to catch audio from native players that bypass Mob.Audio (a game's own AudioTrack). Device testing on a moto g power (2021), Android 11, disproved this: a session-0 Visualizer fails with ERROR_NO_INIT even with RECORD_AUDIO + MODIFY_AUDIO_SETTINGS — global output capture is privileged (signature CAPTURE_AUDIO_OUTPUT, or MediaProjection AudioPlaybackCapture). iOS forbids it by sandbox.

So in-app these probes verify your own audio. True global/foreign-app capture is deferred to a separate MediaProjection-based capture plugin intended as a test-environment dependency (Android-only). For "is a bundled game's audio working" without that plugin, adb shell dumpsys media.audio_flinger (active track + underruns) is the answer.

Device verification (moto g power 2021, Android 11) ✅

dist-RPC into a real build:

  • App boots with the new NIF table (stable pid) — boot-critical for a native-table mismatch.
  • output_status/0%{route: :speaker, volume: 0.2, muted: false, other_audio: false}.
  • output_level(:mob) while a looped tone played → {-34.8, -31.8}; idle / after stop → {:error, :not_playing}.
  • output_level(:mix){:error, :unsupported_on_platform}.
  • Without RECORD_AUDIO granted → {:error, :needs_record_audio}.

How

Native wiring mirrors screenshot/3 + open_settings/1: -export/-nifs/stub in mob_nif.erl, native table entries in mob_nif.zig + mob_nif.m, cacheOptional + null-guard for the app-owned Android bridge methods (drift-safe boot). output_level is a dirty IO NIF. Android returns a length-coded float[] (len 2 = level; len 1 = error code) the NIF maps to atoms; decoding (route codes, :silent/:error) is pure Elixir (decode_status/1, decode_level/1), unit-tested on host. Companion Android Kotlin: GenericJam/mob_new#25.

Host: mob test suite green, formatters clean (mix format, erlfmt, clang-format, zig fmt). Decision record + follow-up plugin scope: decisions/2026-06-29-audio-output-probes.md.

🤖 Generated with Claude Code

Mob can verify visual output in-process via screenshot/3, but had no
equivalent for audio. This adds two read-only probes that answer "is
sound actually coming out right now":

- Mob.Audio.output_status/0 — {volume, muted, route, other_audio}.
  Cheap, no permission. Catches the common silence causes (muted,
  volume 0, dead route). iOS AVAudioSession / Android AudioManager.
- Mob.Audio.output_level/1 — {rms_db, peak_db} | :silent | {:error, _}.
  Reads actual signal energy, the part output_status and `adb dumpsys
  audio` cannot answer. source: :mix (default) taps the global output
  mix so it sees native players that bypass Mob.Audio (e.g. a game's
  own AudioTrack); source: :mob taps Mob.Audio's own player.

Native wiring mirrors screenshot/3 + open_settings/1: -export/-nifs/stub
in mob_nif.erl, native table entries in mob_nif.zig and mob_nif.m, and
cacheOptional + null-guard for the app-owned Android bridge methods so a
drifted MobBridge.kt no-ops instead of crash-looping boot. iOS level-2
uses AVAudioPlayer metering (self-contained); Android uses a Visualizer
on session 0 (needs RECORD_AUDIO). output_level is a dirty IO NIF.

Honest asymmetry: for audio that bypasses Mob.Audio, level-2 works on
Android (global-mix tap) but not iOS (sandbox forbids it) — documented,
not papered over. Decoders are pure Elixir and unit-tested on host; the
Android Kotlin bridge methods ship in the mob_new template (separate PR).

Host: 969 mob tests pass, formatters clean. Device verification (boot +
live level reads) is the gate for the native paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@GenericJam

Copy link
Copy Markdown
Owner Author

Companion Kotlin bridge methods: GenericJam/mob_new#25. Merge together.

Device verification on a moto g power (2021), Android 11, corrected the
design: a session-0 (global output mix) Visualizer fails with ERROR_NO_INIT
for a normal app even with RECORD_AUDIO + MODIFY_AUDIO_SETTINGS — global
output capture is privileged. So the global-mix `:mix` path can't work in
the core framework.

Revised output_level/1:
- default source is now :mob — meters Mob.Audio's own player. iOS via
  AVAudioPlayer metering; Android via a Visualizer on the player's OWN
  audio session (audioPlayer.audioSessionId), which works with RECORD_AUDIO.
- :mix returns {:error, :unsupported_on_platform} on both platforms. True
  global/foreign-app capture is deferred to a separate MediaProjection-based
  plugin (test-env dep) — see the decision doc.

The Android NIF now decodes a length-coded return (float[2] = level;
float[1] = error code 1 unsupported / 2 needs_record_audio / 3 not_playing).

Verified on hardware (dist-RPC into doom_demo):
  output_status      → %{route: :speaker, volume: 0.2, muted: false}
  output_level(:mob) → {-34.8, -31.8} playing; {:error, :not_playing} idle
  output_level(:mix) → {:error, :unsupported_on_platform}
  no RECORD_AUDIO    → {:error, :needs_record_audio}
  app boots cleanly (NIF table OK)

output_status/0 is unchanged and fully verified. Companion template update:
mob_new (own-session Visualizer, no session-0 tap).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@GenericJam

Copy link
Copy Markdown
Owner Author

Review — approve with follow-ups (FFI seam clean, drift-safe)

Traced both new NIFs (audio_output_status/0 ()[F, audio_output_level/1 (Ljava/lang/String;)[F) across every layer — Elixir → mob_nif.erl → zig table/descriptor → Kotlin @JvmStatic → iOS nif table. Arities/descriptors match, dirty flags match. And unlike the input-metering PR #67, this uses cacheOptional + null-guards, so a drifted app bridge degrades gracefully (no boot crash) — nice. decode_status/decode_level host tests are meaningful.

Two semantic gaps worth tracking (both verified in the diff, not blockers):

1. :silent never fires on Android. decode_level({_rms, peak}) when peak <= -120.0 -> :silent, but this PR's own doc notes the Android Visualizer floor is -96 dB. -96 > -120, so pushed silence on Android decodes as a real signal — :silent is effectively iOS-only. A caller using output_level(...) == :silent to detect silence gets a false negative on Android. Suggest raising the threshold above the Android floor (e.g. -95.0, or normalize per-platform) and adding a {-96.0, -96.0} host test (the suite tests {-160,-160}/{-130,-120} but never the Android floor).

2. output_level(:mob) doesn't see play_at. iOS meters g_audio_player (the play/1 player); the sample-accurate play_at path uses separate scheduled players that never get meteringEnabled. Android is analogous (audioPlayer MediaPlayer vs the AudioTrack scheduling path). So after Mob.Audio.play_at, output_level(:mob) reports :not_playing/:not_metering while sound is audibly playing — notably the feature's motivation (an app driving its own AudioTrack) is exactly the path the probe can't see. Suggest metering scheduled players too, or documenting the play/1-only scope.

Minor: the title "verify sound is actually working" is a bit stronger than what two probes measure (signal-present + config-sane, not emission-at-the-speaker) — the module docs are honest about this, so just don't let the title over-set expectations.

Merge with mob_new #25 (Android needs its bridge methods). CI green, base current.

(FFI parity, cacheOptional, the -120/-96 gap, and the g_audio_player/play_at split are confirmed against PR-head; iOS metering values are not device-verified per the PR.)

Integrates mob#67 (input-level metering, now on master) with the output
probes on this branch:
- audio.ex: drop the duplicate decode_level/1 the merge produced (the
  shared helper now lives once; output_level + input_level both use it).
- audio_test.exs: keep decode_status/1 tests; unify decode_level/1 to cover
  atoms from both probes (:not_playing/:needs_record_audio, :not_metering,
  :unsupported_on_platform).
- ios/mob_nif.m: forward-declare g_audio_player so output_level/1 compiles
  (it referenced the play/1 player ~620 lines before its definition — a
  latent iOS build break independent of the merge).

Elixir compiles --warnings-as-errors; 30 audio tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@GenericJam
GenericJam merged commit b316a3c into master Jul 7, 2026
4 checks passed
@GenericJam
GenericJam deleted the feat/audio-output-probes branch July 7, 2026 21:37
GenericJam added a commit that referenced this pull request Jul 7, 2026
A build-time `deps` symlink (→ an absolute local path) was accidentally
staged by `git add -A` in the integration worktree and rode the #54 squash
into master. It's a self-referential absolute-path symlink that breaks any
other checkout. Untrack it; deps/ is regenerated by `mix deps.get` and is
gitignored (/deps/).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@GenericJam GenericJam mentioned this pull request Jul 7, 2026
GenericJam added a commit that referenced this pull request Jul 7, 2026
- Mob.Audio.output_status/0 + output_level/1 — verify sound is actually playing (#54)
- Mob.Audio.start_input_metering/1 + input_level/0 + stop_input_metering/1 — agent ears (#67, MOB-35)

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant