Evaluation harness
How often does the model get it right?
The hand-written SQL behind the business questions is the baseline: it answers all of them by construction. This page measures a language model against it on the same database, by execution match, with intervals instead of a single score. No results are published here: the site has no AI budget, so you run it with your own key and the results stay in your browser. The evaluation design explains the choices.
Run the evaluation with your model
22 questions (20 answerable, 2 not) go to claude-haiku-4-5 one at a time with the same prompt Ask uses. Each query is validated, run in the read-only sandbox and compared with the hand-written answer. One run makes 22 calls billed to your key.
Try the grader yourself (no key needed)
Write your own SQL for any question and it is validated, run and scored exactly as a model's query would be. Nothing is sent anywhere.
How many museum tickets were sold with each payment method for orders placed between 1 July 2019 and 30 June 2020 inclusive? Return the payment method and the number of tickets.
The 22 questions and their gold SQL
The first ten paraphrase the brief's business questions with every parameter stated, so each has one right answer. Ten more cover the rest of the schema, and two ask for data the database does not hold: the right answer there is to decline. All were written before any model was run against them.
- b01-payment-methodsBrief question 1
How many museum tickets were sold with each payment method for orders placed between 1 July 2019 and 30 June 2020 inclusive? Return the payment method and the number of tickets.
Tickets are order-line quantities, not orders; four joins.
Gold SQL and answer
SELECT p.method, sum(l.quantity) AS tickets FROM purchase_order o JOIN payment p ON p.payment_id = o.payment_id JOIN order_line l ON l.order_id = o.order_id JOIN product pr ON pr.product_id = l.product_id WHERE pr.code IN ('TICKET_ONLINE', 'TICKET_ON_SITE') AND date(o.ordered_at) BETWEEN '2019-07-01' AND '2020-06-30' GROUP BY p.method - b02-walkers-entranceBrief question 2
For visits from 30 August 2019 to 29 February 2020 inclusive, which entrance had the most visitors who arrived on foot? Return the entrance name and that number of visitors.
Transport is recorded only on first entries; re-entries must not count.
Gold SQL and answer
SELECT en.name, count(*) AS walkers FROM museum_visit v JOIN entrance en ON en.entrance_id = v.entrance_id WHERE v.transport_mode = 'Walk' AND v.visit_date BETWEEN '2019-08-30' AND '2020-02-29' GROUP BY en.entrance_id ORDER BY walkers DESC LIMIT 1 - b03-audio-guide-shareBrief question 3
What percentage of museum visitors in calendar year 2019 hired at least one audio guide? Round to one decimal place.
A visitor with two hires counts once; integer division gives 0.
Gold SQL and answer
SELECT round(100.0 * count(DISTINCT h.ticket_id) / count(DISTINCT v.ticket_id), 1) AS pct FROM museum_visit v LEFT JOIN audio_guide_hire h ON h.ticket_id = v.ticket_id WHERE v.visit_date BETWEEN '2019-01-01' AND '2019-12-31' - b04-guide-minutesBrief question 4
For every audio guide hired on 6 August 2019, list the device serial number and how many whole minutes it was out (return time minus hire time, rounded to the nearest minute).
Date arithmetic on text timestamps with julianday().
Gold SQL and answer
SELECT serial_number, CAST(round((julianday(returned_at) - julianday(issued_at)) * 1440) AS INTEGER) AS minutes FROM audio_guide_hire WHERE date(issued_at) = '2019-08-06' - b05-wing-routeBrief question 5
Take each ticket's wing scans in time order and merge consecutive scans of the same wing. Which route is most common, and how many tickets followed it? Write the route as the wing names joined by ' > ' (for example 'Denon > Sully').
Needs a window function and an ordered string aggregate; the 2020 design could not answer it.
Gold SQL and answer
WITH scans AS ( SELECT ticket_id, scanned_at, scan_id, wing_id, lag(wing_id) OVER (PARTITION BY ticket_id ORDER BY scanned_at, scan_id) AS previous_wing FROM wing_scan ), routes AS ( SELECT sc.ticket_id, group_concat(w.name, ' > ' ORDER BY sc.scanned_at, sc.scan_id) AS route FROM scans sc JOIN wing w ON w.wing_id = sc.wing_id WHERE sc.previous_wing IS NULL OR sc.previous_wing <> sc.wing_id GROUP BY sc.ticket_id ) SELECT route, count(*) AS tickets FROM routes GROUP BY route ORDER BY tickets DESC LIMIT 1 - b06-all-three-wingsBrief question 6
How many tickets were scanned into all three wings?
Repeat visits to a wing must not count twice.
Gold SQL and answer
SELECT count(*) AS tickets FROM (SELECT ticket_id FROM wing_scan GROUP BY ticket_id HAVING count(DISTINCT wing_id) = 3) - b07-top-exhibitionsBrief question 7
Which ten special exhibitions had the most admissions at the Hall Napoleon door? Return each title and its number of admissions.
Attendance is admissions, not bookings; four joins.
Gold SQL and answer
SELECT e.title, count(h.scan_id) AS admissions FROM exhibition e JOIN exhibition_slot s ON s.exhibition_id = e.exhibition_id JOIN exhibition_booking b ON b.slot_id = s.slot_id JOIN hall_napoleon_scan h ON h.booking_id = b.booking_id GROUP BY e.exhibition_id ORDER BY admissions DESC LIMIT 10 - b08-popular-slotsBrief question 8Order scored
Across all exhibitions and dates, which five 15-minute start times had the most Hall Napoleon admissions? Return the start time and admissions, most first.
A ranking, so order is scored.
Gold SQL and answer
SELECT s.start_time, count(h.scan_id) AS admissions FROM exhibition_slot s JOIN exhibition_booking b ON b.slot_id = s.slot_id JOIN hall_napoleon_scan h ON h.booking_id = b.booking_id GROUP BY s.start_time ORDER BY admissions DESC LIMIT 5 - b09-unused-group-ticketsBrief question 9
Consider online orders that bought 30 or more tickets. In total, how many of the tickets in those orders were never scanned at an entrance?
An anti-join, and the order size lives in order_line.
Gold SQL and answer
SELECT count(*) AS unused FROM purchase_order o JOIN order_line l ON l.order_id = o.order_id JOIN product p ON p.product_id = l.product_id AND p.code = 'TICKET_ONLINE' JOIN ticket t ON t.order_id = o.order_id WHERE o.channel = 'online' AND l.quantity >= 30 AND NOT EXISTS (SELECT 1 FROM entry_scan e WHERE e.ticket_id = t.ticket_id) - b10-winter-morningsBrief question 10
In December, January and February (all years), which weekday from Monday to Friday had the most visitors whose first entry was before 12:00? Return the weekday name (for example 'Tuesday') and the number of visitors.
SQLite numbers weekdays from Sunday = 0; needs a CASE to name them.
Gold SQL and answer
SELECT CASE strftime('%w', visit_date) WHEN '1' THEN 'Monday' WHEN '2' THEN 'Tuesday' WHEN '3' THEN 'Wednesday' WHEN '4' THEN 'Thursday' WHEN '5' THEN 'Friday' END AS weekday, count(*) AS visitors FROM museum_visit WHERE CAST(strftime('%m', visit_date) AS INTEGER) IN (12, 1, 2) AND strftime('%w', visit_date) NOT IN ('0', '6') AND time(first_entry_at) < '12:00:00' GROUP BY strftime('%w', visit_date) ORDER BY visitors DESC LIMIT 1 - x01-tickets-soldExtra
How many museum tickets were sold in total, online and on site?
Sum of quantities, not a count of rows.
Gold SQL and answer
SELECT sum(l.quantity) AS tickets FROM order_line l JOIN product p ON p.product_id = l.product_id WHERE p.code IN ('TICKET_ONLINE', 'TICKET_ON_SITE') - x02-revenue-2019Extra
What was the total revenue in euros from all products for orders placed in 2019? Use quantity times the unit price charged, and round to two decimal places.
Money is stored in cents.
Gold SQL and answer
SELECT round(sum(l.quantity * l.unit_price_cents) / 100.0, 2) AS revenue_eur FROM order_line l JOIN purchase_order o ON o.order_id = l.order_id WHERE strftime('%Y', o.ordered_at) = '2019' - x03-top-languageExtra
Which audio-guide language was chosen most often? Return the language name and the number of hires.
A lookup join for the name.
Gold SQL and answer
SELECT lang.name, count(*) AS hires FROM audio_guide_hire h JOIN language lang ON lang.language_code = h.language_code GROUP BY lang.language_code ORDER BY hires DESC LIMIT 1 - x04-devices-in-serviceExtra
How many audio-guide devices are still in service, that is, have never been retired?
NULL means still in service.
Gold SQL and answer
SELECT count(*) AS devices FROM audio_guide_device WHERE retired_on IS NULL - x05-hires-by-wingExtra
For each wing, how many audio guides were hired there? Return the wing name and the number of hires.
Two wing columns; the hire wing is issued_wing_id.
Gold SQL and answer
SELECT w.name, count(*) AS hires FROM audio_guide_hire h JOIN wing w ON w.wing_id = h.issued_wing_id GROUP BY w.wing_id - x06-busiest-monthExtra
Which calendar month had the most museum visitors? Return the month as 'YYYY-MM' and the number of visitors.
Visitors are first entries, not every scan.
Gold SQL and answer
SELECT strftime('%Y-%m', visit_date) AS month, count(*) AS visitors FROM museum_visit GROUP BY month ORDER BY visitors DESC LIMIT 1 - x07-full-slotsExtra
How many exhibition time slots were fully booked, with no places left?
The slot_availability view answers it directly.
Gold SQL and answer
SELECT count(*) AS full_slots FROM slot_availability WHERE remaining = 0 - x08-cash-shareExtra
What percentage of all payments were made in cash? Round to one decimal place.
Integer division gives 0 without the 100.0.
Gold SQL and answer
SELECT round(100.0 * sum(method = 'cash') / count(*), 1) AS cash_pct FROM payment - x09-re-entriesExtra
How many tickets were scanned at an entrance more than once, that is, re-entered the museum?
Grouping before counting.
Gold SQL and answer
SELECT count(*) AS tickets FROM (SELECT ticket_id FROM entry_scan GROUP BY ticket_id HAVING count(*) > 1) - x10-top-bankExtra
Which bank issued the cards behind the most card payments? Return the bank's name and the number of card payments.
Card details live in a subtype table.
Gold SQL and answer
SELECT fi.name, count(*) AS payments FROM card_payment cp JOIN financial_institution fi ON fi.institution_id = cp.institution_id GROUP BY fi.institution_id ORDER BY payments DESC LIMIT 1 - u01-visitor-ageUnanswerable
What is the average age of visitors who hired an audio guide?
The database records no ages; the right answer is to decline.
- u02-guide-complaintsUnanswerable
Which audio-guide language received the most complaints from visitors?
There is no feedback or complaints data; the right answer is to decline.
Bootstrap seed 20003. Runs and their scores are kept in this browser (IndexedDB); every call is also in the audit log.