This document provides an overview of all ASAP components and links to detailed documentation.

graph TB
    subgraph "Configuration (Offline)"
        U[User] -->|edits| CC[controller-config.yaml]
        CC --> C[asap-planner-rs]
        C -->|streaming_config.yaml| AS[asap-summary-ingest]
        C -->|inference_config.yaml| Q
        AS -->|create pipelines| A
    end

    subgraph "Data Ingestion (Real-time)"
        E[Exporters] -->|metrics| P[Prometheus]
        P -->|remote_write| A[Arroyo]
        A -->|build sketches| A
        A -->|produce| K[Kafka]
    end

    subgraph "Query Execution (Real-time)"
        K -->|consume| Q[QueryEngine]
        G[Grafana] -->|PromQL| Q
        Q -->|results| G
        Q -.->|fallback| P
    end

    subgraph "Experiments (Research)"
        EXP[asap-tools] -->|deploy & run| E
        EXP -->|deploy & run| P
        EXP -->|deploy & run| A
        EXP -->|collect results| EXP
    end

    style C fill:#fff4e1
    style AS fill:#fff4e1
    style A fill:#e1f5ff
    style Q fill:#e1f5ff
    style EXP fill:#f0f0f0

These run continuously to serve queries:

• asap-query-engine - Answers PromQL queries using sketches

  • Consumes sketches from Kafka
  • Implements Prometheus HTTP API
  • Forwards unsupported queries to Prometheus

• Arroyo - Builds sketches from metrics streams

  • Receives Prometheus remote write
  • Executes SQL pipelines
  • Produces sketches to Kafka

These run once to set up the system:

• asap-planner-rs - Determines optimal sketch parameters

  • Analyzes query workload
  • Selects sketch algorithms
  • Generates configs for Arroyo and QueryEngine

• asap-summary-ingest - Creates Arroyo pipelines

  • Reads streaming_config.yaml
  • Renders SQL templates
  • Creates pipelines via Arroyo API

These are used for development and experiments:

• Exporters - Generate synthetic metrics

  • Fake exporters with configurable cardinality
  • Real trace data exporters
  • Performance monitoring exporters

• asap-tools - Experiment orchestration

  • Deploy ASAP to CloudLab
  • Run controlled experiments
  • Collect and analyze results

Performance-critical components written in Rust:

• asap-query-engine - Sub-millisecond query execution
• Arroyo - High-throughput stream processing
• Fake Exporters - Fast metric generation

Configuration and orchestration in Python:

• asap-planner-rs - Query analysis and config generation
• asap-summary-ingest - Pipeline configuration
• asap-tools - Experiment framework
• Python Exporters - Simpler metric generators

asap-query-engine
├── Kafka (runtime) - Consumes sketches
├── Prometheus (runtime, optional) - Fallback queries
└── inference_config.yaml (config) - From asap-planner-rs

Arroyo
├── Prometheus (runtime) - Remote write source
├── Kafka (runtime) - Sketch output
└── SQL pipelines (config) - From asap-summary-ingest

asap-summary-ingest
├── Arroyo (runtime) - Creates pipelines via API
└── streaming_config.yaml (config) - From asap-planner-rs

asap-planner-rs
├── controller-config.yaml (input) - User-provided
├── streaming_config.yaml (output) - For asap-summary-ingest
└── inference_config.yaml (output) - For asap-query-engine

Exporters
└── (standalone, no dependencies)

asap-tools
├── All components (deploys and orchestrates)
└── Hydra configs (experiment specifications)

• asap-query-engine - Query processor deep dive
• Arroyo - Streaming engine + ASAP customizations
• asap-summary-ingest - Pipeline configurator
• asap-planner-rs - Auto-configuration service
• Exporters - Metric generators
• asap-tools - Experiment framework

For implementation details, see READMEs co-located with code:

• asap-query-engine/docs/ - Extensibility guides
• asap-planner-rs/README.md - asap-planner-rs internals
• asap-summary-ingest/README.md - Pipeline config internals
• asap-tools/data-sources/prometheus-exporters/README.md - Exporter implementations
• asap-tools/docs/ - Experiment framework architecture