⚡ Synap

High-Performance In-Memory Key-Value Store & Message Broker

Synap is a modern, high-performance data infrastructure system built in Rust, combining the best features of Redis, RabbitMQ, and Kafka into a unified platform for real-time applications.

Synap provides multiple core capabilities in a single, cohesive system:

• 💾 Key-Value Store - Sharded in-memory storage with TTL, LRU/LFU eviction, and atomic operations
• 🧱 Redis-compatible data structures - Hashes, Lists, Sets, Sorted Sets, Bitmaps, HyperLogLog, Geospatial indexes, and string extensions (APPEND, GETRANGE, SETRANGE, …)
• 📜 Lua Scripting - Server-side scripting with EVAL/EVALSHA and a redis.call() bridge
• 🔄 Transactions - MULTI/EXEC/WATCH/DISCARD — atomic, durable (single WAL group-commit), replicated, and isolated from concurrent writers
• ⏸️ Blocking operations - BLPOP/BRPOP/BRPOPLPUSH and BZPOPMIN/BZPOPMAX with Redis timeout semantics
• 🔍 Cursor scans - SCAN plus HSCAN/SSCAN/ZSCAN with MATCH glob and COUNT
• 🔔 Keyspace notifications - Redis-style notify-keyspace-events (__keyspace@0__:* / __keyevent@0__:*) via Pub/Sub
• 👀 KV Watch - KV.WATCH <pattern> streams value-carrying change events ({ key, event, version, value? }) over SynapRPC push or the /kv/ws WebSocket — a watcher never has to re-GET; available in all six SDKs
• 📨 Acknowledgment Queues - RabbitMQ-style message queues with delivery guarantees, DLQ, priorities, and per-consumer prefetch/QoS
• 📡 Event Streams - Kafka-style partitioned topics with consumer groups, retention policies, and consumer-offset-aware buffering
• 🔔 Pub/Sub Messaging - Topic-based publish/subscribe with wildcard support

• 🏁 Redis 7 parity, measured: driven by the same redis-benchmark binary, Synap matches Redis at -P 1 on every command and sits at 0.85–0.95 pipelined on a single hot key — see Performance
• 📈 Wins on realistic workloads: with a randomized keyspace, Synap beats Redis 7 on SET/INCR/LPUSH by 14–25%, and at 200 connections wins SET at 1.19×
• 🚀 Native SynapRPC protocol: ~3× faster than the RESP3 compatibility path per operation (166k vs 56k rps at -P 1)
• 🧵 Zero-copy hot paths: values live behind shared Arc<[u8]> buffers — parsed once, stored, replied, and replicated by refcount bump, no payload memcpy
• 💾 Efficient Memory: 92MB for 1M keys (vs ~200MB in Redis)
• 🔄 64-Way Sharding: multi-core scalability with lock-free atomic stats
• ⚙️ Async I/O: Built on Tokio for non-blocking operations
• 🗜️ Smart Compression: LZ4/Zstd compression with minimal CPU overhead
• ⚡ SIMD Acceleration: Runtime-dispatched AVX2/NEON/SIMD128 for BITCOUNT, BITOP, PFMERGE — up to 9.5× faster than scalar on HyperLogLog merge

• 🔒 Authentication on every transport - HTTP, RESP3, SynapRPC, and MCP all gate commands behind auth; destructive/admin commands (FLUSHALL, CONFIG, SHUTDOWN, …) additionally require admin
• 🔑 bcrypt password hashing - Constant-time comparison; legacy SHA-512 hashes transparently rehash on next login
• 🛡️ Fine-grained Permissions - Users, API keys (expiration, IP filtering), resource-based ACLs
• 📝 Audit Logging - Track all authentication events (login, API key usage, permission denials)
• 🧯 DoS hardening - Parsers cap client-controlled allocations, bounded pub/sub channels, connection limits + idle timeouts on the binary listeners, no reachable panics
• 🔐 Safe defaults - Binary listeners (RESP3/SynapRPC) bind to loopback by default

• 💾 Full Persistence: WAL + Snapshots covering all datatypes — KV, Hash, List, Set, Sorted Set, Queue, and Stream
• 🔄 Async WAL with group commit: Redis-style batching (10K ops/batch, 100µs window); transactions commit as one atomic fsync
• 🧾 Verified snapshots: CRC64 recomputed on load — corrupt/torn snapshots are rejected, never silently loaded
• 🧠 True maxmemory: a shared budget sums every datatype (not just KV), with per-datatype accounting exposed as metrics
• ⚖️ PACELC Model: PC/EL (Consistency during partition, Latency in normal operation)
• ⏱️ Recovery Time: 1-10 seconds from snapshots + WAL replay

• 🔄 Master-Slave Replication: 1 write master + N read replicas, wired directly from config
  • Every datatype converges — KV, hash, list, set, sorted set, queue, stream — via the same applier the WAL recovery path uses
  • Full sync via snapshot transfer (CRC verified) + partial sync via replication log; replicas joining mid-write-stream lose nothing
  • Works with persistence disabled (replication is decoupled from the WAL)
  • Auto-reconnect with intelligent resync; INFO replication reports live role, replica count, offset, and lag
• ✅ Message Acknowledgment: Guaranteed message delivery with ACK/NACK
• 🔁 Event Replay: Stream history and replay; retention protects events the slowest consumer hasn't read
• 🔀 Manual Failover: Promote replica to master capability
• 🧩 Cluster mode (preview): hash-slot topology (16384 slots), slot migration with rollback, and inter-node quota RPC — initialized from config, disabled by default

• 📈 INFO Command - Redis-style server introspection (server, memory, stats, replication, keyspace)
• 🐌 SLOWLOG - Slow query logging with configurable threshold (default 10ms)
• 💾 MEMORY USAGE - Per-key memory tracking across all data types
• 👥 CLIENT LIST - Active connection tracking and management
• 📊 Prometheus Metrics - process-scoped CPU/memory plus broker-level gauges (per-stream length, consumer-group lag, queue depth) at GET /metrics — see Observability

• 🤖 AI Integration: MCP support for Cursor, Claude Desktop, and AI assistants
• 🌊 StreamableHTTP Protocol: Simple HTTP-based streaming protocol
• 🔌 WebSocket Support: Persistent connections for real-time updates
• 📚 Multi-language SDKs: TypeScript, Python, Rust (with reactive PubSub), Go, PHP, and C# clients with full authentication support
• 📖 Rich Examples: Chat, event broadcasting, task queues, authentication examples, and more

Synap supports three wire transports. All SDKs (Rust, TypeScript, Python, Go, PHP, C#) select the transport via URL scheme — no separate builder options required.

💡 Recommendation — use synap://. SynapRPC is the preferred transport for production workloads: it keeps a persistent multiplexed TCP connection, avoids HTTP framing overhead, and preserves integer/float/bool/bytes types on the wire (no stringification). All data commands — KV, hashes, lists, sets, sorted sets, queues, streams, pub/sub, transactions, scripts, geospatial, HyperLogLog — are supported on every transport (streams and transactional write-queuing landed on the native wires in 1.3.0; queued writes travel as TXQUEUE, see Transports).

Since 1.2.0 the synap:// wire is Thunder — the HiveLLM family's shared binary RPC — on both ends: the server listener and the Rust, TypeScript, Python and C# SDKs run the same protocol implementation, so the two halves of a connection cannot drift. Wire v1 is frozen; a pre-1.2.0 client still interoperates. Every SDK — Rust, TypeScript, Python, C#, Go and PHP — runs the same protocol implementation, verified against one server build by the interop matrix, see docs/thunder-interop-matrix.md.

// TypeScript
const synap = new SynapClient("synap://127.0.0.1:15501");

# Python
client = SynapClient(SynapConfig("synap://127.0.0.1:15501"))

// Rust
let cfg = SynapConfig::new("synap://127.0.0.1:15501");

// PHP
$client = new SynapClient(new SynapConfig("synap://127.0.0.1:15501"));

// C#
var client = new SynapClient(SynapConfig.Create("synap://127.0.0.1:15501"));

⚠️ No silent HTTP fallback. Native transports (synap://, resp3://) raise UnsupportedCommandError for any command not mapped on that transport instead of silently falling back to HTTP. Use http:// if you need full REST access for commands outside the parity matrix.

Additional integration protocols:

• 🤖 MCP (Model Context Protocol): Configurable tools (KV, Hash, List, Set, Queue, Sorted Set) at /mcp endpoint with authentication support
• 🌐 UMICP (Universal Matrix Inter-Communication Protocol): 13 operations via MCP bridge with TLS support

• 📖 Read Scaling: Multiple replica nodes for distributed reads
• 🏠 Event Rooms: Isolated event streams per room/channel
• 🎯 Partitioned Topics: Kafka-style horizontal scaling with multiple partitions
• 👥 Consumer Groups: Coordinated consumption with automatic rebalancing
• 🔀 Topic Routing: Efficient pub/sub with wildcard matching
• 🔗 Connection Pooling: Client-side connection management

┌─────────────────────────────────────────────────────────────────────┐
│                          Synap Server                               │
├──────────────────┬──────────────────────┬───────────────────────────┤
│  HTTP/REST       │  SynapRPC (TCP)      │  RESP3 (TCP)              │
│  :15500          │  :15501 (msgpack)    │  :6379 (Redis text)       │
├──────────────────┴──────────────────────┴───────────────────────────┤
│  server/handlers/  (17 modules)                                     │
│  kv · hash · list · set · sorted_set · hll · bitmap · geospatial    │
│  queue · stream · pubsub · script · websocket · partition · cluster │
├─────────────────────────────────────────────────────────────────────┤
│  protocol/resp3/command/        protocol/synap_rpc/dispatch/        │
│  kv · collections · advanced   kv · collections · advanced          │
├─────────────────────────────────────────────────────────────────────┤
│  core/  (data stores)                                               │
│  kv_store/  bitmap/  hash  list  set  sorted_set  hll  geospatial   │
│  queue  stream  pubsub  transactions  scripting                     │
├─────────────────────────────────────────────────────────────────────┤
│  Replication Log (Append-Only)  ·  WAL + Snapshots persistence      │
├─────────────────────────────────────────────────────────────────────┤
│  Master Node                    Replica Nodes (Read-Only)           │
└─────────────────────────────────────────────────────────────────────┘

Since v1.0.0 the workspace is split into focused crates under crates/ (Vectorizer/Nexus layout). Layers go Foundation → Core → Features → Presentation; higher layers depend on lower, never the reverse.

crates/
├── synap-core/src/core/     # In-memory data engine (leaf crate — no server deps)
│   ├── kv_store/            # Sharded KV store (TTL, eviction, atomic LRU)
│   ├── hash · list · set · sorted_set · bitmap · hyperloglog · geospatial
│   └── queue · stream · pubsub · partition · transaction · consumer_group
├── synap-server/src/        # HTTP/WS/MCP/UMICP + protocol dispatch
│   ├── server/handlers/     # REST handlers (kv, hash, list, set, queue, stream, pubsub, …)
│   ├── protocol/resp3/      # RESP3 parser/writer + listener + command dispatcher
│   ├── protocol/synap_rpc/  # SynapRPC command catalog + config + listener (wire: thunder)
│   ├── persistence/         # WAL (async group-commit) + snapshots
│   ├── replication/         # master / replica
│   └── auth/ · hub/         # users/api-keys/ACL + HiveHub.Cloud multi-tenant
├── synap-cli/               # Command-line client
└── synap-migrate/           # Migration utilities

sdks/rust/src/               # Rust SDK — Thunder's client under the hood
└── transport/               # SynapRPC / RESP3 / HTTP transports + command mappers

The binary RPC wire layer is not in this repository. It is Thunder (thunder-rpc), the HiveLLM family's shared implementation, which both the server and the Rust SDK depend on — so the two ends of the wire cannot drift.

Rust library consumers: synap_server::core::* is now synap_core::* (umbrella re-exports kept on synap_server for transition), and the former synap-protocol crate is gone — see the CHANGELOG for the type-by-type migration to thunder-rpc.

From GitHub Releases (Recommended):

Pre-built binaries for Linux (x86_64), macOS (Intel and Apple Silicon), and Windows (x86_64) are published on the GitHub Releases page.

# Example (Linux x86_64) — replace <version> with the latest release
wget https://github.com/hivellm/synap/releases/download/v<version>/synap-linux-x86_64.tar.gz
tar -xzf synap-linux-x86_64.tar.gz
cd synap-linux-x86_64
./synap-server --config config/config.yml

🐳 Docker:

Quick Start (Docker Hub):

# Pull and run latest image
docker pull hivehub/synap:latest
docker run -d \
  --name synap \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  hivehub/synap:latest

# Check health
curl http://localhost:15500/health

Build Locally:

# Clone and build
git clone https://github.com/hivellm/synap.git
cd synap
docker build -t hivehub/synap:latest .

# Run container
docker run -d \
  --name synap-server \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  hivehub/synap:latest

With Authentication:

docker run -d \
  --name synap-server \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  -e SYNAP_AUTH_ENABLED=true \
  -e SYNAP_AUTH_REQUIRE_AUTH=true \
  -e SYNAP_AUTH_ROOT_USERNAME=admin \
  -e SYNAP_AUTH_ROOT_PASSWORD=SecurePassword123! \
  -e SYNAP_AUTH_ROOT_ENABLED=true \
  hivehub/synap:latest

Multi-Architecture Build:

# Build and push multi-arch images (AMD64 + ARM64)
./scripts/docker/docker-publish.sh 1.3.1

# Or using PowerShell
.\scripts\docker\docker-publish.ps1 1.3.1

Docker Compose:

# Use docker-compose for replication setup
docker-compose up -d

# With authentication (set environment variables)
export SYNAP_AUTH_ENABLED=true
export SYNAP_AUTH_REQUIRE_AUTH=true
export SYNAP_AUTH_ROOT_USERNAME=admin
export SYNAP_AUTH_ROOT_PASSWORD=SecurePassword123!
docker-compose up -d

Available Images:

• hivehub/synap:latest - Latest stable release
• hivehub/synap:<version> - Specific version (e.g. hivehub/synap:1.3.1)
• Supports linux/amd64 and linux/arm64 architectures

📖 For detailed Docker documentation, see DOCKER_README.md

🛠️ From Source:

# Clone repository
git clone https://github.com/hivellm/synap.git
cd synap

# Build from source (requires Rust nightly 1.92+)
cargo build --release

# Run server
./target/release/synap-server --config config/config.yml

See Development Guide for detailed build instructions.

# Start server (default port 15500)
synap-server

# Key-Value Operations
curl -X POST http://localhost:15500/kv/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe", "ttl": 3600}'

curl http://localhost:15500/kv/get/user:1

# Queue Operations
curl -X POST http://localhost:15500/queue/publish \
  -d '{"queue": "tasks", "message": "process-video", "priority": 1}'

curl http://localhost:15500/queue/consume/tasks

# Event Stream
curl -X POST http://localhost:15500/stream/publish \
  -d '{"room": "chat-room-1", "event": "message", "data": "Hello!"}'

# Pub/Sub
curl -X POST http://localhost:15500/pubsub/publish \
  -d '{"topic": "notifications.email", "message": "New order"}'

# With Authentication (if enabled)
curl -X POST http://localhost:15500/kv/set \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe"}'

# Or with API Key
curl -X POST http://localhost:15500/kv/set \
  -H "Authorization: Bearer sk_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe"}'

Authentication is disabled by default for development. Enable it for production:

Features:

• ✅ Root User Management - Configurable root user with full permissions
• ✅ User Management - Create, delete, enable/disable users
• ✅ API Key Management - Generate, revoke, and manage API keys with expiration
• ✅ Fine-grained Permissions - Resource-based permissions (RabbitMQ-style)
• ✅ Basic Auth & Bearer Token - Support for both authentication methods
• ✅ Audit Logging - Track all authentication events (login, API key usage, permission denials)
• ✅ Password Validation - Configurable password requirements (length, complexity)
• ✅ IP Filtering - Restrict API keys to specific IP addresses

Via Config File (config/config.yml):

auth:
  enabled: true
  require_auth: true
  root:
    username: "root"
    password: "your_secure_password"
    enabled: true

Via Docker Environment Variables:

docker run -d -p 15500:15500 \
  -e SYNAP_AUTH_ENABLED=true \
  -e SYNAP_AUTH_REQUIRE_AUTH=true \
  -e SYNAP_AUTH_ROOT_USERNAME=root \
  -e SYNAP_AUTH_ROOT_PASSWORD=your_secure_password \
  synap:latest

Using Authentication:

# Basic Auth
curl -u root:password http://localhost:15500/kv/get/user:1

# Bearer Token (API Key)
curl -H "Authorization: Bearer sk_XXXXX..." http://localhost:15500/kv/get/user:1

# Query Parameter
curl "http://localhost:15500/kv/get/user:1?api_key=sk_XXXXX..."

See Authentication Guide for complete details.

Synap supports master-slave replication for high availability and read scaling.

# Start 1 master + 3 replicas
docker-compose up -d

# Master available at: localhost:15500
# Replica 1 at: localhost:15510
# Replica 2 at: localhost:15520
# Replica 3 at: localhost:15530

Master Node Configuration (config/config-master.yml):

server:
  host: "0.0.0.0"
  port: 15500

replication:
  enabled: true
  role: "master"
  replica_listen_address: "0.0.0.0:15600"  # 15600 = replication (SynapRPC uses 15501)
  heartbeat_interval_ms: 1000
  max_lag_ms: 10000
  buffer_size_kb: 256
  replica_timeout_secs: 30

persistence:
  enabled: true
  wal:
    enabled: true
    path: "./data/wal/synap.wal"
  snapshot:
    enabled: true
    directory: "./data/snapshots"

Replica Node Configuration (config/config-replica.yml):

server:
  host: "0.0.0.0"
  port: 15500

replication:
  enabled: true
  role: "replica"
  master_address: "master:15600"  # Master's replication port
  heartbeat_interval_ms: 1000
  max_lag_ms: 10000
  buffer_size_kb: 256
  auto_reconnect: true
  reconnect_delay_ms: 5000

persistence:
  enabled: true
  wal:
    enabled: true
    path: "./data/wal/synap.wal"
  snapshot:
    enabled: true
    directory: "./data/snapshots"

Start Nodes:

# Terminal 1: Start master
synap-server --config config/config-master.yml

# Terminal 2: Start replica 1
synap-server --config config/config-replica.yml

# Terminal 3: Start replica 2
synap-server --config config/config-replica.yml

Write to Master:

# All writes go to master
curl -X POST http://localhost:15500/kv/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:100", "value": "Alice", "ttl": 3600}'

Read from Replicas (Load Balancing):

# Read from replica 1 (eventually consistent, ~5ms lag)
curl http://localhost:15510/kv/get/user:100

# Read from replica 2
curl http://localhost:15520/kv/get/user:100

# Read from replica 3
curl http://localhost:15530/kv/get/user:100

Monitor Replication Status:

# Check replication health on master
curl http://localhost:15500/health/replication

# Check replication status on replica
curl http://localhost:15501/health/replication

• Master Reads: Strongly consistent (immediate)
• Replica Reads: Eventually consistent (~5-10ms lag typical)
• Write Durability: Writes confirmed after master commit
• Replication: Asynchronous to replicas
• Lag Monitoring: Real-time offset tracking

See docs/specs/REPLICATION.md for complete replication documentation.

For detailed Docker deployment guide, see:

• DOCKER_README.md - Complete Docker Hub documentation with examples
• docs/specs/DOCKER_DEPLOYMENT.md - Advanced deployment guide

Use event streams for room-based messaging with message history and guaranteed delivery.

Leverage acknowledgment queues for distributed task processing with retry logic.

Utilize key-value store as a high-speed cache with TTL support.

Implement pub/sub for system-wide notifications and event distribution.

Use queues for reliable inter-service messaging with delivery guarantees.

• Language: Rust (Edition 2024, workspace of focused crates: synap-core, synap-server, synap-cli, synap-migrate)
• Runtime: Tokio (async/await)
• Web Framework: Axum
• Storage: 64-way sharded stores (ahash) with Arc<[u8]> shared values; radix trie for pub/sub topic routing
• Serialization: serde (JSON, MessagePack)
• Protocols: SynapRPC + RESP3 + HTTP/StreamableHTTP + WebSocket + MCP + UMICP

• User Guide - Complete getting started guide (Installation, Quick Start, Operations)
• Admin Guide - Operations handbook (Deployment, Monitoring, HA, Security)
• Tutorials - 8 practical tutorials (Chat, Queues, Caching, Pub/Sub)

• Architecture - System architecture and components
• Configuration - Complete configuration reference
• CLI Guide - Synap CLI usage and commands
• Transports - SynapRPC / RESP3 / HTTP command-parity matrix
• Transactions - MULTI/EXEC durability, replication, and isolation
• KV Watch - Value-carrying change streams, modes, version ordering, fan-out cost
• Replication - Setup, sync semantics, and monitoring
• Memory Accounting - maxmemory across all datatypes
• Observability - Prometheus metrics reference

• Authentication - Complete auth guide (users, roles, API keys, ACL)
• Network Limits - Connection caps, idle timeouts, parser bounds

• REST API - Complete REST API documentation
• OpenAPI Spec - OpenAPI 3.0 specification (YAML/JSON)
• StreamableHTTP - StreamableHTTP protocol
• MCP Integration - Model Context Protocol
• UMICP Integration - UMICP protocol

• Redis vs Synap - Live redis-benchmark head-to-head
• Benchmarks - All benchmark suites and results
• Queue Concurrency - Zero-duplicate guarantees
• Testing Strategy - Test coverage and approach

• Development Guide - Setup and contribution guide
• Design Decisions - Technical choices
• Roadmap - Development roadmap and timeline
• Project DAG - Component dependencies and implementation order
• Deployment - Production deployment
• Packaging - Distribution packages

• Key-Value Store - Sharded storage system
• Queue System - Message queues with ACK
• Event Stream - Room-based broadcasting
• Pub/Sub - Topic-based messaging
• Replication - Master-slave architecture

• SDK Index - All six SDKs: transports, module coverage, quick start
• TypeScript SDK - Node.js and browser support
• Python SDK - Async/sync Python client
• Rust SDK - Native Rust client library
• Go SDK - Standalone repository, vendored here as a submodule
• PHP SDK - Standalone repository, vendored here as a submodule
• C# SDK - .NET 8+ client library

• Real-Time Chat - Multi-room chat application
• Event Broadcasting - System-wide events
• Task Queue - Distributed task processing
• Pub/Sub Pattern - Notification system

Both servers driven by the same redis-benchmark binary over RESP (release build, containers on one Docker network, persistence and auth off on both sides). Full methodology and history: docs/benchmarks/redis-vs-synap.md

Single hot key (Redis's best case — sharding buys Synap nothing):

Randomized 1M keyspace (the realistic shape — Synap's shards parallelize, Redis stays serial):

Native SynapRPC (Synap's own protocol; RESP3 exists for compatibility) is ~3× faster per operation than the RESP3 path — 166–170k rps vs ~56k at -P 1 — and ~2.8× faster than HTTP/JSON on a SET+GET round-trip.

Memory: 92 MB for 1M keys vs ~200 MB in Redis (54% less).

Tests: 1,800+ across the workspace • Benchmarks: 13 criterion suites + synap-bench load generator

Legend: ✅ Implemented | 🔄 In Progress | ❌ Not Available

Apache License 2.0 - See LICENSE for details.

See DEVELOPMENT.md for development setup and contribution guidelines.