Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ MYSQL_VERSION_84=8.4
MYSQL_VERSION_93=9.3
MYSQL_VERSION_96=9.6

# Dolt (MySQL-compatible, version-controlled database)
DOLT_VERSION=latest

# Individual PostgreSQL Versions
POSTGRES_VERSION_14=14
POSTGRES_VERSION_15=15
Expand All @@ -24,6 +27,7 @@ DB_PORT_MYSQL80=3306
DB_PORT_MYSQL84=3307
DB_PORT_MYSQL93=3308
DB_PORT_MYSQL96=3309
DB_PORT_DOLT=3310

# Database Ports (PostgreSQL)
DB_PORT_POSTGRES14=5414
Expand Down
106 changes: 106 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# DBDiff — Project Guidelines

## What This Is

Automated database schema & data diff tool that generates SQL migration files. Compares two databases and produces UP + DOWN SQL. Built-in migration runner with versioned history tracking.

Supports MySQL 8.0–9.6, PostgreSQL 14–18, SQLite 3, plus MySQL-compatible variants (MariaDB, Aurora, PlanetScale, Vitess, TiDB, Dolt) and Supabase.

## Architecture

```
src/
DB/ — Adapters (MySQL, Postgres, SQLite), schema introspection, data diffing
Diff/ — 27 diff object models (AddTable, AlterTableChangeColumn, InsertData, CreateView, CreateTrigger, CreateRoutine, etc.)
SQLGen/ — SQL generation: Dialect/ (MySQL, Postgres, SQLite), DiffToSQL/ (27 generators)
Migration/ — Commands (Symfony Console), Runner, Config, Format/ (Native, Flyway, Liquibase, Laravel)
Params/ — CLI parameter parsing (CLI flags → config file → defaults)
Exceptions/ — Exception hierarchy
```

Key flow: `DiffCommand` → `DBDiff` orchestrator → `DiffCalculator` (schema + data) → `SQLGenerator` → `MigrationGenerator` → output file.

**Design patterns**: Factory (AdapterFactory, DialectRegistry, FormatRegistry), Strategy (adapters, dialects, formats), Command (Symfony Console), Registry.

**Namespace**: `DBDiff\{Module}` with PSR-4 autoloading.

## Build & Test

```bash
composer install # Install dependencies
composer run build:phar # Compile PHAR binary (uses box.json)

# Tests — use the wrapper script:
./scripts/run-tests.sh # Full test run
./scripts/run-tests.sh --unit # Unit tests only
./scripts/run-tests.sh --postgres # PostgreSQL E2E
./scripts/run-tests.sh --sqlite # SQLite E2E
./scripts/run-tests.sh --specific <method> # Single test method
./scripts/run-tests.sh --record # Record new baselines
```

**Local runs use Podman** (not Docker). The wrapper script handles this. For direct PHPUnit invocations:
```bash
podman run --rm -v "$(pwd):/app:Z" -w /app php:8.4-cli vendor/bin/phpunit ...
```

**CI matrix**: 5 PHP versions (8.1–8.5) × 4 MySQL versions, Dolt, PostgreSQL 14–18, SQLite. See `.github/workflows/tests.yml`.

## Conventions

- **PHP 8.1+** minimum. No PHP attributes used yet — stick to docblocks.
- **One class per file**, matching classname.
- **Test baselines**: `tests/expected/` contains golden output files. Run `--record` to regenerate after intentional output changes.
- **Test fixtures**: `tests/fixtures/` for schema/data setups, `tests/end2end/` for scenario-based E2E tests.
- **Config resolution order**: CLI flags → `dbdiff.yml` → `.dbdiff.yml` → `dbdiff.yaml` → `.dbdiff`
- **DSN URLs**: Full database URLs supported (`postgres://user:pass@host:5432/db`), parsed by `DsnParser`.
- **Commit format**: `type(scope): subject` — Angular convention. Types: feat, fix, docs, style, refactor, perf, test, chore.

## Key Gotchas

- Each `DiffToSQL/` generator implements both `getUp()` and `getDown()` — always update both when modifying SQL generation.
- SQLite has limited ALTER TABLE — the adapter uses table-rebuild strategy. Test SQLite separately.
- `InsertDataSQL` and `DeleteDataSQL` emit explicit column-name lists (not `INSERT INTO t VALUES`).
- The `_dbdiff_migrations` table tracks migration history — never diff this table.
- Supabase mode (`--supabase`) sets `driver=pgsql`, `sslMode=require`, and enables dual-write to `supabase_migrations.schema_migrations`.
- MySQL adapter's `normalizeCreateStatement()` strips DEFINER, ALGORITHM, SQL SECURITY, and trailing semicolons from view/trigger/routine definitions.
- PostgreSQL `DROP TRIGGER` requires `ON table` — handled by `PostgresDialect::dropTrigger()`.
- SQLite has no stored procedures/functions — `getRoutines()` returns `[]`.
- DiffSorter places DROP view/trigger/routine BEFORE table ops, CREATE/ALTER AFTER data ops.

## Docs

- [README.md](../README.md) — Usage, supported databases, download
- [DOCKER.md](../DOCKER.md) — Docker setup
- [docs/SUPABASE.md](../docs/SUPABASE.md) — Supabase integration
- [docs/CROSS_DB_TRANSLATION.md](../docs/CROSS_DB_TRANSLATION.md) — Cross-DB SQL strategies
- [docs/diff-algorithms.md](../docs/diff-algorithms.md) — Algorithm details
- [docs/DB-ecosystem-compatibility.md](../docs/DB-ecosystem-compatibility.md) — Variant support matrix
- [.github/CONTRIBUTING.md](CONTRIBUTING.md) — PR guidelines, commit format

## NPM Distribution

`packages/@dbdiff/cli/` contains a TypeScript/Node wrapper that bundles pre-compiled PHAR binaries for 8 platforms.

```bash
cd packages/@dbdiff/cli
npm install
npm test # Vitest unit tests
npm run test:integration # Integration tests
npm run build # Compile TypeScript
```

See `packages/@dbdiff/cli/README.md` for full details.

## Definition of Done

Every branch or piece of independent work MUST satisfy all of the following before it is considered complete:

- **Minimal dependencies**: Make the simplest fix or feature possible. Do NOT add external production dependencies unless unavoidable. Dev dependencies are fine only if: actively maintained open source, MIT or Apache 2.0 license, regular releases, and many contributors.
- **DRY code**: Don't Repeat Yourself. Before implementing any functionality, check existing utils, helpers, and config in the codebase. Extract repeated logic into reusable methods or classes.
- **No magic values**: Key config, numbers, and settings must not be hardcoded in source files. Extract them to named constants or config.
- **Low complexity**: No single file should carry too much responsibility. Extract reusable classes and utilities with their own dedicated unit tests.
- **Tests**: Add or update unit tests AND e2e tests. Register new test suites in the GHA workflow (`.github/workflows/tests.yml`) so they run in CI.
- **Local verification**: Run all relevant unit and e2e tests locally via Podman (or Docker if installed) and confirm they pass before treating the work as done.
- **Docs updated**: Update `README.md`, inline docblocks, and any relevant *existing* file under `docs/` to reflect the change. Include a usage example for any user-facing change. Do NOT create new documentation files unless explicitly instructed to do so.
- **Clean commits**: Never force-add files or folders excluded by `.gitignore`. Do not mention gitignored paths or any part of their file contents in commit messages or PR descriptions.
68 changes: 65 additions & 3 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,68 @@ jobs:
- name: Run comprehensive tests
run: ./scripts/run-tests.sh --testsuite DBDiff,Unit

# ── Dolt (MySQL-compatible, version-controlled database) ────────────────────
test-dolt:
name: PHP ${{ matrix.php-version }} / Dolt
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
php-version: ['8.3', '8.4']

env:
DB_HOST: 127.0.0.1
DB_PORT: 3306
DB_ROOT_PASSWORD: rootpass
DB_USER: dbdiff
DB_PASSWORD: dbdiff
DBDIFF_ENGINE: dolt

services:
dolt:
image: dolthub/dolt-sql-server:latest
env:
DOLT_ROOT_PASSWORD: rootpass
DOLT_ROOT_HOST: "%"
MYSQL_DATABASE: diff1
MYSQL_USER: dbdiff
MYSQL_PASSWORD: dbdiff
ports:
- 3306:3306
options: >-
--health-cmd="bash -c '(echo > /dev/tcp/localhost/3306) 2>/dev/null'"
--health-interval=10s
--health-timeout=5s
--health-retries=30

steps:
- uses: actions/checkout@v4

- name: Wait for Dolt to be query-ready
run: |
for i in $(seq 1 30); do
if mysql -h 127.0.0.1 -u root -prootpass -e "SELECT 1" >/dev/null 2>&1; then
echo "Dolt is ready for queries"
break
fi
echo "Waiting for Dolt... ($i/30)"
sleep 2
done

- name: Setup PHP
uses: shivammathur/setup-php@44454db4f0199b8b9685a5d763dc37cbf79108e1 # v2
with:
php-version: ${{ matrix.php-version }}
extensions: pdo, pdo_mysql, pdo_sqlite, mysqli, json
coverage: none

- name: Install dependencies
run: composer install --prefer-dist --no-progress

- name: Run comprehensive tests against Dolt
run: ./scripts/run-tests.sh --testsuite DBDiff,Unit

# ── SQLite standalone (no DB service needed) ─────────────────────────────────
test-sqlite:
name: PHP ${{ matrix.php-version }} / SQLite e2e
Expand Down Expand Up @@ -308,9 +370,9 @@ jobs:
\$url = 'postgres://postgres:[email protected]:5432/postgres';
\$parsed = \DBDiff\Migration\Config\DsnParser::parse(\$url);
echo 'Driver: ' . \$parsed['driver'] . PHP_EOL;
echo 'Host: ' . \$parsed['server']['host'] . PHP_EOL;
echo 'Port: ' . \$parsed['server']['port'] . PHP_EOL;
echo 'DB: ' . \$parsed['db'] . PHP_EOL;
echo 'Host: ' . \$parsed['host'] . PHP_EOL;
echo 'Port: ' . \$parsed['port'] . PHP_EOL;
echo 'DB: ' . \$parsed['name'] . PHP_EOL;

// Verify actual PDO connection
\$dsn = 'pgsql:host=127.0.0.1;port=5432;dbname=postgres';
Expand Down
3 changes: 3 additions & 0 deletions DOCKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,11 +111,14 @@ Podman runs containers as your own user by default. If port binding below 1024 f
- `cli-php84-mysql93` - PHP 8.4 with MySQL 9.3
- `cli-php83-postgres16` - PHP 8.3 with PostgreSQL 16
- `cli-php84-postgres16` - PHP 8.4 with PostgreSQL 16
- `cli-php83-dolt` - PHP 8.3 with Dolt (MySQL-compatible)
- `cli-php84-dolt` - PHP 8.4 with Dolt (MySQL-compatible)

### Database Services
- `db-mysql80` - MySQL 8.0 (accessible on localhost:3306)
- `db-mysql84` - MySQL 8.4 (accessible on localhost:3307)
- `db-mysql93` - MySQL 9.3 (accessible on localhost:3308)
- `db-dolt` - Dolt (accessible on localhost:3310)
- `db-postgres16` - PostgreSQL 16 (accessible on localhost:5432)

### PHPMyAdmin Services
Expand Down
38 changes: 32 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
## Features

- Compares two databases (local or remote) and generates SQL migrations automatically
- Diffs schema changes, data changes, or both — with deterministic, predictable output
- Diffs tables, views, triggers, stored procedures/functions, and data — with deterministic, predictable output
- Up and down SQL generated in the same file
- Built-in migration runner: `migration:up`, `down`, `status`, `validate`, `repair`, `baseline`
- Supports MySQL, PostgreSQL, and SQLite via `--driver`
Expand Down Expand Up @@ -91,6 +91,7 @@ The databases below work with DBDiff's existing drivers with no code changes. **
| Vitess / VTGate | MySQL wire protocol via VTGate |
| Percona XtraDB Cluster | MySQL-compatible; Galera replication metadata ignored |
| TiDB | MySQL-compatible; default port 4000 |
| [Dolt](https://github.com/dolthub/dolt) | MySQL-compatible, version-controlled; **CI-tested** |

### PostgreSQL-compatible — `--driver=pgsql`

Expand Down Expand Up @@ -135,10 +136,10 @@ The quickest way to get started is to download a pre-built release directly from
| [Pre-built binary](#pre-built-binaries) | ✅ Yes | Quickest start — zero dependencies |
| [PHAR](#phar) | ✅ Yes | Single portable file; requires PHP ≥ 8.1 |
| [npm](#npm) | ✅ Yes (via registry) | Node.js projects or CI pipelines |
| [Docker](#docker) | — | Isolated environments or testing |
| [Docker / Podman](#docker--podman) | — | Isolated environments, CI, or no local PHP |
| [Composer (source)](#composer-source-install) | — | Contributing to DBDiff or PHP integration |

> **PHP requirement:** Pre-built binaries, npm packages, and Docker images bundle PHP 8.3 — no system PHP needed. The PHAR and Composer installs require **PHP ≥ 8.1** on your system.
> **PHP requirement:** Pre-built binaries, npm packages, and Docker/Podman images bundle PHP 8.3 — no system PHP needed. The PHAR and Composer installs require **PHP ≥ 8.1** on your system.


## Pre-built Binaries
Expand Down Expand Up @@ -188,19 +189,30 @@ dbdiff --version
To build a PHAR locally from source, see [Building a PHAR](#building-a-phar).


## Docker
## Docker / Podman

Pre-built multi-arch images (linux/amd64 + linux/arm64) are published to GHCR on every release.
Pre-built multi-arch images (linux/amd64 + linux/arm64) are published to GHCR on every release. Both Docker and [Podman](https://podman.io/) are fully supported — use whichever is available on your system. No local PHP installation is required.

### Pull and run (no build required)

**Docker:**
```bash
docker pull ghcr.io/dbdiff/dbdiff
docker run --rm ghcr.io/dbdiff/dbdiff --version
docker run --rm ghcr.io/dbdiff/dbdiff --driver=mysql \
--server1=user:pass@host:3306 server1.mydb:server1.mydb
```

**Podman** (drop-in replacement — commands are identical):
```bash
podman pull ghcr.io/dbdiff/dbdiff
podman run --rm ghcr.io/dbdiff/dbdiff --version
podman run --rm ghcr.io/dbdiff/dbdiff --driver=mysql \
--server1=user:pass@host:3306 server1.mydb:server1.mydb
```

> **Podman on Linux** runs rootless by default — no daemon required. Install via your package manager: `sudo apt install podman` (Debian/Ubuntu) or `brew install podman` (macOS).

### Image variants

| Tag pattern | Registry | Description |
Expand All @@ -212,14 +224,15 @@ docker run --rm ghcr.io/dbdiff/dbdiff --driver=mysql \

```bash
# Slim image (requires dist/dbdiff.phar — run `vendor/bin/box compile` first)
# Replace 'docker' with 'podman' if using Podman
docker build -f docker/Dockerfile.slim -t dbdiff:slim .
docker run --rm dbdiff:slim --version

# Full image (Composer install from source — no PHAR needed)
docker build -f docker/Dockerfile -t dbdiff:full .
```

See [DOCKER.md](DOCKER.md) for cross-version testing, Podman usage, and start.sh flags.
See [DOCKER.md](DOCKER.md) for cross-version testing, Podman setup, and start.sh flags.


## Composer Source Install
Expand Down Expand Up @@ -524,6 +537,19 @@ Comparisons run in this order:
- Detects differences in column count, name, type, collation or attributes
- New columns in the source are added to the target

### Views
- Detects created, dropped, and altered views across source and target
- ALTER = DROP IF EXISTS + CREATE with the new definition

### Triggers
- Detects created, dropped, and altered triggers
- PostgreSQL DROP TRIGGER includes the required ON table clause

### Stored Procedures / Functions
- Detects created, dropped, and altered routines (MySQL and PostgreSQL)
- SQLite has no stored procedures — routines are skipped automatically
- MySQL definitions are normalized (DEFINER, ALGORITHM, SQL SECURITY stripped)

### Data
- Compares table storage engine, collation, and row count
- Records changed rows and missing rows per table
Expand Down
54 changes: 54 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -394,8 +394,62 @@ services:
interval: 5s
start_period: 40s

# Dolt (MySQL-compatible, version-controlled database)
db-dolt:
image: docker.io/dolthub/dolt-sql-server:latest
restart: always
ports:
- '${DB_PORT_DOLT}:3306'
environment:
DOLT_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
DOLT_ROOT_HOST: "%"
MYSQL_DATABASE: ${DB_NAME}
MYSQL_USER: ${DB_USER}
MYSQL_PASSWORD: ${DB_PASSWORD}
healthcheck:
test: ["CMD-SHELL", "bash -c '(echo > /dev/tcp/localhost/3306) 2>/dev/null'"]
timeout: 20s
retries: 15
interval: 5s
start_period: 30s

cli-php83-dolt:
depends_on:
db-dolt:
condition: service_healthy
build:
context: .
dockerfile: docker/Dockerfile
args:
PHP_VERSION: ${PHP_VERSION_83}
environment:
- DB_HOST=db-dolt
- DBDIFF_ENGINE=dolt
volumes:
- .:/usr/src/dbdiff
- /usr/src/dbdiff/vendor
command: tail -f /dev/null

cli-php84-dolt:
depends_on:
db-dolt:
condition: service_healthy
build:
context: .
dockerfile: docker/Dockerfile
args:
PHP_VERSION: ${PHP_VERSION_84}
environment:
- DB_HOST=db-dolt
- DBDIFF_ENGINE=dolt
volumes:
- .:/usr/src/dbdiff
- /usr/src/dbdiff/vendor
command: tail -f /dev/null

# PHPMyAdmin services
phpmyadmin-mysql80:

depends_on:
db-mysql80:
condition: service_healthy
Expand Down
Loading
Loading