A modern Python 3.13+ microservices platform for transforming the complete Discogs music database into powerful, queryable knowledge graphs and analytics engines.

🚀 Quick Start | 📖 Documentation | 🎯 Features | 💬 Community

Discogsography transforms monthly Discogs data dumps (~11.3GB compressed XML) into:

• 🔗 Neo4j Graph Database: Navigate complex music industry relationships
• 🐘 PostgreSQL Database: High-performance queries and full-text search
• 🔍 Interactive Explorer: Graph visualisation, trends, and path discovery
• 📊 Real-time Dashboard: Monitor system health and processing metrics
• 🎵 MusicBrainz Enrichment: Cross-reference with MusicBrainz for metadata, relationships, and external links

Perfect for music researchers, data scientists, developers, and music enthusiasts who want to explore the world's largest music database.

graph TD
    S3[("🌐 Discogs S3<br/>Data Dumps")]
    MB[("🎵 MusicBrainz<br/>JSONL Dumps")]

    subgraph Pipeline ["Data Pipeline"]
        EXT[["⚡ Extractor"]]
        RMQ{{"🐰 RabbitMQ"}}
        GRAPH[["🔗 Graphinator"]]
        TABLE[["🐘 Tableinator"]]
    end

    subgraph MBPipeline ["MusicBrainz Enrichment"]
        BGRAPH[["🧠 Brainzgraphinator"]]
        BTABLE[["🧬 Brainztableinator"]]
    end

    subgraph Storage ["Storage"]
        NEO4J[("🔗 Neo4j")]
        PG[("🐘 PostgreSQL")]
        REDIS[("🔴 Redis")]
    end

    subgraph Services ["User-Facing Services"]
        API[["🔐 API"]]
        EXPLORE[["🔍 Explore"]]
        DASH[["📊 Dashboard"]]
        INSIGHTS[["📈 Insights"]]
    end

    S3 --> EXT --> RMQ
    MB --> EXT
    RMQ --> GRAPH --> NEO4J
    RMQ --> TABLE --> PG
    RMQ --> BGRAPH --> NEO4J
    RMQ --> BTABLE --> PG

    API --- NEO4J & PG & REDIS
    EXPLORE --- API
    INSIGHTS -.-> API
    INSIGHTS --- PG & REDIS
    DASH -.- RMQ & NEO4J & PG & REDIS

    style S3 fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    style MB fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    style EXT fill:#ffccbc,stroke:#d84315,stroke-width:2px
    style RMQ fill:#fff3e0,stroke:#e65100,stroke-width:2px
    style NEO4J fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    style PG fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    style REDIS fill:#ffebee,stroke:#b71c1c,stroke-width:2px
    style GRAPH fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style TABLE fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    style BGRAPH fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style BTABLE fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    style API fill:#e3f2fd,stroke:#0d47a1,stroke-width:2px
    style EXPLORE fill:#e8eaf6,stroke:#283593,stroke-width:2px
    style DASH fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    style INSIGHTS fill:#fff9c4,stroke:#f57f17,stroke-width:2px

See Architecture Overview for detailed diagrams covering data pipeline, service communication, and message queue structure.

• ⚡ High-Speed Processing: ~130–480 records/second end-to-end throughput per data type with Rust-based extractor
• 🔄 Smart Deduplication: SHA256 hash-based change detection prevents reprocessing
• 📈 Handles Big Data: Processes 19M+ releases, 10M+ artists across ~11.3GB compressed XML
• 🔁 Auto-Recovery: Automatic retries with exponential backoff and dead letter queues
• 🐋 Container Security: Non-root users, read-only filesystems, dropped capabilities
• 📝 Type Safety: Full type hints with strict mypy validation and Bandit security scanning
• ✅ Comprehensive Testing: Unit, integration, and E2E tests with Playwright
• 🚀 Query Performance: 249x overall query performance optimization across 88 endpoints (PRs #175–#184), plus configurable data quality rules for extraction validation (#187) — see Recent Improvements

# Clone and start all services
git clone https://github.com/SimplicityGuy/discogsography.git
cd discogsography
docker-compose up -d

# Access the dashboard
open http://localhost:8003

See the Quick Start Guide for prerequisites, local development setup, and environment configuration.

• 🐛 Bug Reports: GitHub Issues
• 💡 Feature Requests: GitHub Discussions
• 💬 Questions: Discussions Q&A
• 📚 Full Documentation: docs/README.md

This project is licensed under the MIT License — see the LICENSE file for details.

Some other projects working with the monthly Discogs data dump.

• DiscoDOS
• disco-quick
• discogs-load

• 🎵 Discogs for providing the monthly data dumps
• 🎵 MusicBrainz for the open music encyclopedia and twice-weekly JSONL dumps
• 🚀 uv for blazing-fast package management
• 🔥 Ruff for lightning-fast linting
• 🐍 The Python community for excellent libraries and tools
• 🦀 The Rust community for excellent libraries and amazing performance