SignalMine

SignalMine turns unstructured internet discussions into structured, evidence-backed market intelligence.

Portfolio: Case study · Metrics (125 pytest tests collected) · Author

The product definition and constraints are authoritative in docs/. The canonical data contracts are the JSON Schemas in schemas/.

Validate the core pipeline on a real GitHub issue:

git clone <repository-url>
cd painminingengine
python -m venv .venv
source .venv/bin/activate
pip install -e .
cp .env.example .env

Set these in .env:

Then run:

signalmine sources

signalmine fetch https://github.com/kubernetes/kubernetes/issues/12345

signalmine process https://github.com/kubernetes/kubernetes/issues/12345

If process returns a structured Complaint JSON, the collector → normalizer → extractor → validator pipeline is working. Database, API server, and Reddit credentials can come later.

fetch and process accept a URL and route to the correct collector automatically. No source-specific subcommands are required.

signalmine fetch https://github.com/org/repo/issues/42
signalmine process https://github.com/org/repo/issues/42

Check what is available in your environment:

signalmine sources

Example output:

Available Sources

✓ GitHub
✓ File
○ Reddit (not configured)
○ Hacker News (planned)

All collectors implement SourceCollector.fetch(target) -> RawDiscussion and return dictionaries matching schemas/raw_discussion.json.

GitHubCollector fetches public or token-authenticated GitHub issues via the REST API.

Unauthenticated GitHub API access works for public repositories but hits rate limits quickly. Set GITHUB_TOKEN before running repeated tests.

The current implementation provides:

• source-agnostic RawDiscussion contract and SourceCollector interface
• GitHub and Reddit collectors with URL-based routing
• deterministic RawDiscussion-to-NormalizedDocument conversion
• Instructor-backed extraction into the complaint contract
• direct synchronous collector → normalizer → extractor → validator orchestration
• insert-only SQLAlchemy persistence for every pipeline stage
• Typer commands for stage-by-stage JSON workflows
• golden-dataset extraction metrics and Markdown reports
• FastAPI endpoints for processing, complaint reads, health, and Prometheus metrics

It deliberately contains no authentication, embeddings, clustering, opportunity scoring, workers, or frontend.

Reddit collection requires a Reddit API application. Until credentials are approved and configured, use GitHub as the default validation path.

Create a script app at reddit.com/prefs/apps, then add credentials to .env. Once configured, Reddit URLs work through the same commands:

signalmine fetch https://www.reddit.com/r/devops/comments/abc123/title/
signalmine process https://www.reddit.com/r/devops/comments/abc123/title/

RedditCollector maps PRAW submissions to RawDiscussion. Legacy Reddit envelope JSON (schemas/raw_reddit.json) is still accepted by FileCollector and coerced automatically.

DispatchingNormalizer routes RawDiscussion captures to the correct source normalizer and returns one object matching schemas/normalized_document.json.

The transformation removes Markdown syntax, collapses whitespace deterministically, converts timestamps to UTC ISO 8601, and stores original text under metadata.original_text. No AI inference or persistence occurs in this stage.

ExtractionEngine.extract(normalized) accepts a schema-valid normalized post and returns a strict Pydantic Complaint matching schemas/extraction.json.

Collector → Normalizer → Extractor → Validator → Complaint

process <url> runs the complete flow for GitHub or Reddit URLs. The CLI does not persist results unless storage is injected explicitly (the REST API does).

Persistence requires PostgreSQL, DATABASE_URL, and Alembic migrations. These are needed for the API server, not for the CLI quick start above.

DATABASE_URL=postgresql+psycopg://... alembic upgrade head
uv run uvicorn backend.app.api:create_app --factory

Install the project, then run signalmine --help.

Examples:

signalmine fetch https://github.com/kubernetes/kubernetes/issues/12345
signalmine process https://github.com/kubernetes/kubernetes/issues/12345 --output complaint.json
signalmine normalize --input raw.json --output normalized.json
signalmine extract --input normalized.json --output complaint.json

Python 3.11 or newer is required.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest

backend/app/  Python package boundaries
docs/         Product, architecture, data model, and evaluation contracts
prompts/      Versioned extraction prompt policy
schemas/      Canonical JSON Schema contracts