Skip to content

Add dataset catalog, data loaders, and data preparation tooling for versioned datasets - #6

Open
cto-new[bot] wants to merge 1 commit into
masterfrom
feat-data-workflow-dataset-catalog-yolo-classification-prep-scripts
Open

Add dataset catalog, data loaders, and data preparation tooling for versioned datasets#6
cto-new[bot] wants to merge 1 commit into
masterfrom
feat-data-workflow-dataset-catalog-yolo-classification-prep-scripts

Conversation

@cto-new

@cto-new cto-new Bot commented Nov 23, 2025

Copy link
Copy Markdown

Summary

Introduce a dataset catalog workflow and end-to-end data tooling to register, load, and preprocess versioned datasets for detection and classification tasks, integrated with StreetScenePipeline.

Details

  • Implement dataset catalog (src/data/catalog.py) to store versioned datasets with labels, splits, and provenance hashes in data/datasets.yaml
  • Add YOLO dataset loader (src/data/yolo_dataset.py) and classification dataset loader (src/data/classification_dataset.py) with preprocessing hooks
  • Develop shared augmentation pipelines (src/data/transforms.py) including YOLO mosaics, HSV transforms, and classification policies toggleable via config
  • Add scripts/data_prepare.py to ingest raw data, convert to YOLO/classification formats, generate dataset manifests, and update catalog provenance
  • Update StreetScenePipeline.prepare_data to resolve dataset entries from the catalog and emit task-appropriate targets
  • Document the data lifecycle (acquire → preprocess → register → use) in docs and README
  • Extend training/evaluation scripts to support catalog-based datasets while preserving legacy direct-path usage
  • Note: No breaking changes introduced

Warning: Task VM test is not passing, cto.new will perform much better if you fix the setup

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants