Issues

• Five of the eleven dev databases differ between dev.zip and minidev.zip

  bird-bench/mini_dev#50 · Filter by author ivermin1123 opened on Sep 7, 2026

• BIRD dev (bird_sql_dev_20251106): 23 golds still cut at a tie, sort NULL first, or sort numbers as text

  bird-bench/mini_dev#49 · Filter by author ivermin1123 opened on Sep 7, 2026

• PostgreSQL: q1473 scores differently on two runs of evaluation_ex.py against the same database

  bird-bench/mini_dev#48 · Filter by author ivermin1123 opened on Sep 7, 2026

• evaluation_ex.py crashes with ZeroDivisionError after executing every query when a difficulty bucket is empty (e.g. the train split) — results lost

  bird-bench/mini_dev#47 · Filter by author christian-gheorghe opened on Sep 5, 2026

• Train split: 372 of 9,428 gold queries cannot run against the shipped train databases (gold-as-prediction scores 0.9605; retail_world schema mismatch)

  bird-bench/mini_dev#46 · Filter by author christian-gheorghe opened on Sep 5, 2026

• Dev set: 10 gold/evidence annotation defects with executed evidence (129, 1113, 646, 635, 1199, 271, 1006/1011/1014, 1279)

  bird-bench/mini_dev#45 · Filter by author christian-gheorghe opened on Sep 5, 2026

• --diff_json_path is an INPUT (JSONL difficulty labels), not an output; a missing file crashes after the full evaluation has run

  bird-bench/mini_dev#44 · Filter by author christian-gheorghe opened on Sep 5, 2026

• Two dev gold queries exceed the default --meta_time_out 30.0: dev has a hard ceiling of 0.9987 and the ceiling depends on the grader's hardware

  bird-bench/mini_dev#43 · Filter by author christian-gheorghe opened on Sep 5, 2026

• A malformed prediction entry is silently scored against the "financial" database instead of raising

  bird-bench/mini_dev#42 · Filter by author christian-gheorghe opened on Sep 5, 2026

• evaluation_ex.py pairs predictions to gold POSITIONALLY and ignores the keys — a sort_keys=True prediction file scores 0.0013 with no error

  bird-bench/mini_dev#41 · Filter by author christian-gheorghe opened on Sep 5, 2026

• In fact, gt file obtained through gmail account is broken

  bird-bench/mini_dev#35 · Filter by author jamesunnc opened on May 22, 2026
  10

• Boundary-case data coverage for numeric predicates

  bird-bench/mini_dev#31 · Filter by author tohpinren opened on Jan 18, 2026

• Support for Arabic Language

  bird-bench/mini_dev#28 · Filter by author mhammadpk opened on Nov 30, 2025