Stanford's annual AI Index compiles chatbot data it did not generate

The conversation

Stanford's Institute for Human-Centered AI, HAI, publishes an annual compilation of data on AI research, industry and policy called the AI Index. The current instalment, the 2026 AI Index report, is hosted alongside the project's ai-index landing page, both described here as retrieved on 16 September 2026 rather than tied to a single dated announcement, since each yearly edition supersedes the last on the same site.

What the documents show

The project's own landing page states the report is 'a collaborative effort led by the AI Index Steering Committee, an interdisciplinary group of experts from across academia and industry', and lists named supporting and research partners including Google, OpenAI, the National Science Foundation and McKinsey among others. The 2026 edition's own page lists its chapters, including one titled Technical Performance and another titled Responsible AI, alongside separate chapters on the economy, science and medicine, so a reader can go to the chapter most relevant to a chatbot or agent question rather than a chapter on, say, AI in medicine. The landing page states the 2026 edition's headline finding is a widening gap between what AI systems 'can do' and how prepared organisations and policy are 'to manage it', pointing to accelerating capability and investment against slower governance.

The system boundary

The Index describes itself as a compilation project, not a research lab producing its own primary measurements: its role is to gather, standardise and present data that other named organisations, including AI labs, evaluation groups and researchers, originally collected or published. Because of that structure, a specific benchmark figure appearing inside the Index's Technical Performance chapter is properly attributed to whichever lab or evaluation body first reported it, not to Stanford HAI's own research, and the boundary between the Index's own analysis and its source data is the compilation itself: HAI selects, synthesises and contextualises figures rather than generating the underlying benchmark runs.

Where it fails

Because each yearly edition is published on the same living page as the last, a reader citing 'the AI Index' without naming a year risks citing an outdated or superseded edition; the current site foregrounds the 2026 edition, but editions dating back to 2017 remain listed and reachable. The Index's own reliance on named external sources also means a figure's currency and quality are only as good as those sources are, and the Index's compilation role does not include independently re-running or auditing the benchmarks it reports. A reader should trace a specific chatbot or agent claim in the Index back to the specific named source the report itself cites, rather than treating the Index as the original evidence.

• Which named source does this Index chapter attribute a specific chatbot or agent figure to, and has that source been checked directly?
• Which edition year is being cited, and does a newer edition supersede it?
• Does the compiled figure come from a vendor's self-reported benchmark or from an independent evaluation body?

The Index's own value, on its own description, is bringing scattered figures on chatbots and agents into one comparable place each year, not generating new evidence of its own.