Paper: AIRun, Building MCP Tools From Agent Failures

Summary

AIRun is an MCP server that lets an AI agent query the observability data for eBay's messaging platform, so on-call engineers can investigate incidents in plain language. We didn't design its tools up front. We shipped a minimal version with detailed failure logging, watched what the agent got wrong in real investigations, and changed the tools in response. The paper describes that loop as a five-phase method and reports six months of production results.

Highlights

• A tool description is an error-prevention surface. The biggest single gain, missing partition filters going from 75% of queries to 5%, came from rewriting one tool description to show correct and incorrect usage and what the mistake costs. No code changed.
• Agents fail quietly. A query without the filter timed out, and neither the agent nor we could tell a bad query from a slow cluster. Every response now carries a correlationId, a failureStage and recommendations, which brought debugging a failed call from hours to under a minute.
• Two agent sessions, one loop. One agent does real on-call work. When it hits a problem, it writes the failure up as a structured request file. A second agent implements the fix as a pull request that a human reviews and merges. That produced 15 merged PRs in 4 weeks for 32 hours of human time, and most of the changes were documentation, not code.
• Tool calls have a timeout, and real queries run longer. A three-stage async design (useful error, then submit and poll, then distributed) was added one stage at a time as timeout failures piled up. Before it, a timed-out query came back empty and the agent reported "no drop events found" when there were thousands.
• "Can't do that" goes stale. A memory file said an endpoint didn't exist after it had shipped. The agent now re-checks claims like that, which costs a few seconds.
• The limits are stated. Success is measured from server logs, the case studies aren't a random sample, and everything ran on one model and agent harness.

The paper

Scroll inside the frame to read all 13 pages. Download the PDF (500 KB) for selectable text, or get it from Zenodo.

@misc{apsunde2026airun,
  title     = {AIRun: A Production Framework for Failure-Driven MCP Tool Evolution},
  author    = {Apsunde, Abhijeet and Wali, Prasad and Son, Paul and Malipeddy, Sai Charan},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23027657},
  url       = {https://doi.org/10.5281/zenodo.23027657},
  note      = {Preprint}
}