Problem
The profiling.query_engine: true option in config.yaml was originally implemented for the Python query engine using py-spy. Since the query engine was rewritten in Rust, this flag is completely non-functional:
-
remote_monitor.py raises NotImplementedError — when --profile_query_engine is set and the Rust query engine process keyword is detected (either bare-metal or container), the code explicitly raises NotImplementedError("Profiling for Rust query engine is not implemented yet"). The experiment aborts.
-
do_profiling in the Rust engine is a dead config field — engine_config.rs declares and parses do_profiling: bool from the YAML config, but no Rust code ever reads or acts on it.
-
py-spy is Python-only — even if the guard were removed, main_prometheus_client.py would try to run py-spy record --pid <pid> against a Rust binary, producing garbage output. py-spy understands CPython stack frame internals and cannot profile native binaries.
Context
- The Arroyo engine (also Rust) already has working flamegraph profiling in
remote_monitor.py via ~/.cargo/bin/flamegraph --pid <pid> — the exact same tool that would work for the Rust query engine.
- The Rust workspace
Cargo.toml has no [profile.release] overrides, so release binaries are built without debug symbols, making flamegraph output unreadable without a Cargo.toml change.
- Container mode adds extra complexity (multi-stage Docker build strips symbols); bare-metal mode (
use_container.query_engine: false) sidesteps this entirely since the binary is run directly from target/release/.
Impact
Setting profiling.query_engine: true crashes the experiment immediately. The flag is effectively dead config with no indication to the user that it no longer works.
Problem
The
profiling.query_engine: trueoption inconfig.yamlwas originally implemented for the Python query engine usingpy-spy. Since the query engine was rewritten in Rust, this flag is completely non-functional:remote_monitor.pyraisesNotImplementedError— when--profile_query_engineis set and the Rust query engine process keyword is detected (either bare-metal or container), the code explicitly raisesNotImplementedError("Profiling for Rust query engine is not implemented yet"). The experiment aborts.do_profilingin the Rust engine is a dead config field —engine_config.rsdeclares and parsesdo_profiling: boolfrom the YAML config, but no Rust code ever reads or acts on it.py-spyis Python-only — even if the guard were removed,main_prometheus_client.pywould try to runpy-spy record --pid <pid>against a Rust binary, producing garbage output.py-spyunderstands CPython stack frame internals and cannot profile native binaries.Context
remote_monitor.pyvia~/.cargo/bin/flamegraph --pid <pid>— the exact same tool that would work for the Rust query engine.Cargo.tomlhas no[profile.release]overrides, so release binaries are built without debug symbols, making flamegraph output unreadable without a Cargo.toml change.use_container.query_engine: false) sidesteps this entirely since the binary is run directly fromtarget/release/.Impact
Setting
profiling.query_engine: truecrashes the experiment immediately. The flag is effectively dead config with no indication to the user that it no longer works.