← Back to the journal

DataFocus · Journal

Is Text-to-SQL Enough for Business Users? Raw SQL Generation vs Governed Ontology Queries

Public enterprise benchmarks show raw text-to-SQL still fails on most real warehouse tasks. The gap is mostly missing business definitions. Here is how definition-bound queries that show their SQL compare, and when plain text-to-SQL is fine.

For exploratory questions on a small, clean, well-named schema, a text-to-SQL tool is often enough. For numbers that business users will repeat in meetings, it usually is not: public enterprise benchmarks show models still fail most real warehouse tasks, and the main cause is missing business definitions rather than weak SQL. What closes the gap is a governed layer that fixes the definitions once, binds every AI query to them, and shows the SQL and data behind each answer so an analyst can check it.

What do public benchmarks show about enterprise text-to-SQL?

Academic text-to-SQL looked close to solved a few years ago. The original Spider dataset (Yu et al., 2018) used 200 databases, and by 2023 the DAIL-SQL prompt-engineering method (Gao et al., 2023) reported 86.6% execution accuracy on the Spider leaderboard. Enterprise benchmarks tell a different story.

Benchmark (year)What it testsReported result
Spider 2.0 paper, Lei et al. (Nov 2024, v2 Mar 2025)632 real-world enterprise workflow problems; databases "often containing over 1,000 columns"An o1-preview code-agent framework "successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD"
BEAVER (arXiv, v3 May 2026)Private enterprise data warehouses: 9,128 question-SQL pairs, 812 tablesState-of-the-art agentic frameworks with GPT-5.2 at 10.8% accuracy; 30.1% with subtask hints
data.world knowledge-graph benchmark, Sequeda et al. (Nov 2023)One enterprise insurance SQL schemaGPT-4 zero-shot on SQL: 16%; over a knowledge-graph representation (ontology plus mappings): 54%

Two cautions. First, the Spider 2.0 project page leaderboard has moved since launch, so quote any figure with its date; the 2024 numbers above describe the state of the field then, not today's best system. Second, these are research benchmarks, not your warehouse. They are useful for one thing: showing that the jump from a tidy demo schema to an enterprise schema costs a lot of accuracy, across several independent teams and years.

Why do definitions, not the model, cause most wrong numbers?

Look at what an enterprise question hides. "What was revenue by region last quarter?" requires the system to know:

  • which of several amount columns is "revenue" (gross, net of returns, net of tax);
  • which order statuses count (shipped, invoiced, paid);
  • whether "quarter" is calendar or fiscal;
  • which table holds the authoritative region for a customer when CRM and ERP disagree;
  • what currency the amounts are in.

None of that is in the column names. A model that reads the schema has to guess, and a guess that produces valid SQL produces a confident wrong number. The data.world result above points the same way: the same model on the same questions did much better once an ontology and mappings described what the tables meant.

A recent vendor post makes the argument directly. Collate (April 2026) reports that adding its semantic layer raised Sonnet 4.5 execution accuracy on a Spider 2.0-Snow subset from 10.8% to 76.5%. That is a single vendor's own study and it does not report question counts or variance, so treat it as direction, not proof.

The semantic-layer idea is not new to BI. dbt Labs' documentation (as documented on November 2026) describes moving metric definitions "out of the BI layer and into the modeling layer" so different business units work from the same definitions. The newer question is whether AI queries are held to those definitions too, and whether the user can see that they were.

What does a governed alternative look like in practice?

Using DataFocus OT as one concrete example, here is the mechanism. The point is the pattern; other platforms implement parts of it differently.

1. Definitions are set once and shared. A business definition is set once and used by search analysis, dashboards, control panels, the ontology agent and external agents, which all get the same figures. Users can also add business definitions and terminology for FocusGPT, the natural-language assistant, through "My Knowledge Base" (content, keywords, type, category and an enabled flag), and administrators manage system-wide knowledge, including FocusGPT's confirmed question-to-keyword mappings, under Admin > Knowledge.

2. Natural language is translated into a visible, editable statement, not straight into hidden SQL. In Data Search, users type column names and keywords such as "Sales by Region". If the input is a full question instead, the system asks FocusGPT to parse it into a search statement and shows the parsed statement with a Revert action. The analyst sees exactly what the question became before trusting the result.

3. The SQL is shown. The result's right-hand rail has a "Query SQL Statement" panel showing the SQL the search actually ran, so a user can check how a figure was calculated.

4. Each query returns a receipt. Every query returns the SQL that ran, the tables it used and a data fingerprint, and the AI follows the objects and relationships defined in the ontology instead of guessing. For agents connected over MCP, the OT Flash agent manual describes SQL results returned through evidenceRows with an execution receipt, completeness and truncation information and an evidenceRowsSha256 hash, and says zero-row, failed or truncated results must not be described as complete statistics. The same manual says business meaning such as currency, period, order status and tie-breaking must be clarified before ranking, and units must not be guessed from field names.

5. The AI is labeled as AI. FocusGPT's Smart Q&A page warns that content is AI-generated, and the manual advises checking important conclusions in Data Search.

In DataFocus's published product brief, a demo built on sample data shows a business user in Data Search and Claude Code connected over MCP getting matching department totals, with the agent's answer citing the SQL and the evidenceRowsSha256 receipt field (OT product brief). That is a demonstration with built-in sample data, not a customer deployment.

Side by side

Question an analyst should askTypical raw text-to-SQL toolGoverned, definition-bound query
Where does "revenue" come from?Inferred from schema and prompt each timeA definition set once and reused
Can I see what my question became?Varies; often SQL onlyParsed statement shown, with revert
Can I see the SQL?Usually yesYes
Can I prove which rows produced the number?RarelyReceipt with tables and a data fingerprint
Do the dashboard and the AI agree?Not guaranteedSame definitions feed both
Setup effortLow: point at a databaseHigher: someone must model and maintain definitions

When is plain text-to-SQL fine?

Raw text-to-SQL is a reasonable choice, and often the better one, when:

  • the user can read SQL and will check it before using the number;
  • the schema is small, documented and has one obvious meaning per column;
  • the work is exploratory and no figure will be published or reused;
  • you are a data engineer prototyping a query, not a business user reporting a KPI;
  • there is no one available to own a definitions layer. A governed layer that nobody maintains drifts, and stale definitions can be worse than none because they look authoritative.

The governed approach has costs. Someone has to model the objects and definitions and keep them current. It also adds a platform. In DataFocus's case, OT Flash uses Apache Doris as both its compute engine and data warehouse, aimed at one business domain, while OT Heavy builds on Hadoop, Hive, Trino and Iceberg for organizations with an existing big-data platform. Either one is a platform someone has to deploy and run. If your need is one analyst asking ad-hoc questions of a Postgres replica, a text-to-SQL tool plus SQL review is cheaper.

Evaluation questions before you pick a tool

Use these on any vendor, including the one above. Run them against your own schema, not the demo database.

  1. Ambiguity test. Ask a question with two plausible meanings ("revenue last quarter"). Does the tool ask, apply a stored definition, or silently pick one?
  2. Consistency test. Ask the same question in the AI assistant and in the dashboard. Do the numbers match, and if not, can the tool explain why?
  3. Transparency test. Can a business user see what the question was translated into, and the SQL, without admin rights?
  4. Evidence test. Does the answer say which tables and rows it used, and does it flag truncated or empty results instead of reporting them as totals?
  5. Ownership test. Where are definitions stored, who can change them, and is a change recorded?
  6. Wide-schema test. Point it at your widest real schema, not a curated view. Benchmarks such as Spider 2.0 suggest this is where accuracy drops.

A decision rule that works for most teams: if a number will leave the analyst's screen, it should come from a definition someone owns and arrive with the SQL that produced it. If it will not, plain text-to-SQL is fine.

Next step

To see how the two DataFocus editions differ in engine and scope, read the edition comparison on the download page. For the query, receipt and definition mechanisms described above, the OT product brief is the shortest official summary. Both editions are free: installers and release notes, including system requirements, are published on the download page and in the OT Flash and OT Heavy GitHub repositories.