LLM API Benchmark v2

LLMs are bad at writing working integration code. Here's why:

1. Trained on outdated documentation

For instance, in tasks requiring OpenAI's newer Responses API with file inputs, most LLMs attempted legacy multi-part uploads instead of passing the new input_file content object. This is a particularly frustrating failure mode because it looks "reasonable" in code but doesn't run.

2. Missing training data for niche and long-tail systems

For industry-specific or lesser-known systems (e.g., legacy systems or startups with solid but sparsely discussed APIs), LLMs guessed endpoints, misread pagination styles, or confused core entities (e.g., "workspace" vs. "user" in CRMs like Attio). Without examples in training data, LLMs hallucinate.

3. Can't debug autonomously

LLMs failed to generate working multi-step flows, even if individual steps are "easy" (Download HTML → convert via Jinja API to Markdown → analyze with OpenAI). LLMs struggled to understand failures and failed to auto-correct or debug follow-on steps; superglue's stepwise planning and self-healing isolated and repaired errors correctly.

4. Don't handle Auth reliably

While modern services with API keys or bot tokens were fine, LLMs massively struggled with legacy and edge cases (e.g., OAuth1 or multi-stephandshakes), because they can't persist and reason over intermediate auth state.

5. APIs were built for humans, not agents

When API design is informal or inconsistent, LLMs invent plausible but wrong parameters (API expects Query Parameter, but LLM adds parameters to the request body) or path shapes. superglue retrieves the exact OpenAPI spec for an endpoint while it builds a tool and thereby knows what query parameter to pass.