The headline number in BI-Agent is that frontier LLMs score under 50% on end-to-end business intelligence, and dedicated NL2SQL systems manage only 6–17% on the same tasks. That gap is the whole story. The hard part of BI was never writing the final query — it’s the plumbing before it: finding the right tables, building join relationships, transforming data, all the tedious prep a business user does by hand today before they can ask their actual question.
This matches what I see in production. When an agent fails on a structured-data question, the failure almost never lives in the SQL. It lives three steps earlier — in picking the wrong table or missing a join — and by then the model is confidently answering the wrong question. BI-Agent’s move is to stop treating BI as one prompt and decompose it into subtasks — search, join, transform — with specialized data-management methods behind each stage, then post-train on trajectories synthesized from real BI projects. Tool-augmented reasoning buys up to 40 points over a vanilla LLM; SFT plus RL on top adds roughly another 30.
The lesson generalizes well past dashboards. The frontier-model-as-oracle framing keeps losing to orchestration-plus-grounding on any task where the real work is assembling context out of messy enterprise data. What I’d want next from the BI-Bench work is an ablation that cleanly separates how much of that 40-point gain comes from the decomposition versus the domain post-training — because one of those you can adopt tomorrow, and the other needs a training pipeline most teams don’t have. Which half is really carrying the result?