Data Platform:

This project is an end-to-end batch data platform built to practice production-oriented Data Engineering patterns. The pipeline extracts the public thelook_ecommerce dataset from Google BigQuery, bulk-loads it into Snowflake, transforms it through a Bronze → Silver → Gold architecture with dbt, and orchestrates the complete workflow with Apache Airflow + Cosmos.

flowchart LR
    BQ["Google BigQuery<br/>thelook_ecommerce"]
    PY["Python Ingestion<br/>Parquet + COPY INTO"]
    B["Snowflake<br/>BRONZE"]
    S["dbt<br/>SILVER"]
    G["dbt<br/>GOLD"]
    BI["Analytics-ready<br/>Star Schema"]
    BQ --> PY --> B --> S --> G --> BI
    AF["Airflow + Cosmos"]
    AF -. orchestrates .-> PY
    AF -. orchestrates .-> S
    AF -. orchestrates .-> G

Instead of row-by-row inserts, the ingestion follows a bulk-load pattern:

BigQuery → Pandas → Parquet → Snowflake Stage → COPY INTO

Data is first loaded into a temporary table before replacing the target data, making pipeline reruns idempotent and controlling duplicate records.

Airflow generates one ingestion task per configured source table:

ingest_distribution_centers
ingest_inventory_items
ingest_order_items
ingest_products
ingest_users

These tasks can execute independently and in parallel, with 2 retries configured for transient failures. Only after ingestion completes successfully does the transformation layer begin.

The transformation layer follows a medallion architecture. Silver standardizes the raw source data through stg_* views and Gold exposes an analytics-ready dimensional model.

Data quality is enforced through dbt tests covering: Primary-key uniqueness and not-null, referential integrity, accepted values, numeric ranges and business rules (eg. non-negative gross profit)

Snowflake infrastructure is reproducible through versioned SQL scripts covering:

Warehouse → Database → Schemas → Roles → RBAC → Tables

data-platform/
├── config/             # Central pipeline configuration
├── infrastructure/     # Snowflake provisioning & RBAC
├── ingestion/          # BigQuery → Snowflake ingestion
├── transformation/     # dbt staging, marts, tests & macros
├── orchestration/      # Airflow / Cosmos
└── documentation/      # Architecture & DAG assets

The Gold layer provides an analytics-ready dimensional model centered on sales.

flowchart LR
    DC[(distribution_centers)] --> SDC[stg_distribution_centers] --> DDC[dim_distribution_centers]
    INV[(inventory_items)] --> SINV[stg_inventory_items] --> DINV[dim_inventory_items]
    OI[(order_items)] --> SOI[stg_orders_items] --> F[fct_sales]
    P[(products)] --> SP[stg_products] --> DP[dim_products]
    SP --> F
    U[(users)] --> SU[stg_users] --> DU[dim_users]

fct_sales contains order-item-level sales information plus some macros calculations (gross_profit, gross_margin).

Install all the requirements.

Run the SQL scripts in infrastructure/ in order.

gcloud auth application-default login

Create a .env file:

SNOWFLAKE_PASSWORD=<your_password>

Update config/config.py with your GCP and Snowflake configuration.

cd orchestration
astro dev start

Open the Airflow UI and trigger dataplatform_pipeline.

Run infrastructure/999_destroy.sql to remove the Snowflake resources.