How Zapier Cut Spec Hallucinations to Under 1% With AI

Zapier's team couldn't measure how often its spec-gen LLM hallucinated. A seven-metric evaluation framework made it visible — dropping hallucinated paths from 26% to under 1%.

26% to <1%

Drop in hallucinated endpoint paths

Implementation Time

Chris Taylor

CEO & Co-Founder

Fractional AI

Connect ↓

The Challenge

Zapier's LLM-powered spec-generation system automatically produced OpenAPI specifications from API documentation, but its performance was opaque. The team could not measure accuracy, quantify the impact of pipeline changes, or identify where to prioritize improvements.

What They Built

Fractional AI built a comprehensive LLM evaluation framework with seven distinct metrics measuring correctness and hallucination rates across endpoint paths and field properties, paired with iterative experimentation to optimize the four-step pipeline.

The pipeline scrapes API docs to markdown, lists endpoints, extracts relevant documentation, and generates OpenAPI specs. Evaluations ran on Braintrust against curated, manually reviewed test data. Winning experiments included targeted prompt engineering, passing extra context (name, HTTP verb, URL) between stages, model selection (GPT-4o default, Claude Sonnet 3.5 for extraction), and prompt caching for cost savings.

AI Role

The framework does not generate specs itself; it scores them. Seven metrics measure correctness and hallucination rates across endpoint paths and field properties, running each experiment against curated test data on Braintrust. This let the team compare prompt changes, context passing, and model choices (GPT-4o by default, Claude Sonnet 3.5 for extraction) and keep only the changes that measurably improved accuracy.

AI Model

GPT-4o, Claude Sonnet 3.5, Braintrust evaluation platform, curated OpenAPI specs and Postman collections as test data

Infrastructure

Braintrust evaluation platform • Existing four-step OpenAPI spec-generation pipeline • Test set of curated, manually reviewed OpenAPI specs and Postman collections

Integration Points

Evaluation harness wired into the spec-generation pipeline on Braintrust • Extra context (endpoint name, HTTP verb, URL) passed between pipeline stages • Prompt caching layer for cost reduction

Impact

Endpoint hallucinations 26% to <1%

Hallucinated endpoint paths dropped from 26% to less than 1%.

Field type accuracy ~2x

Field type accuracy improved nearly 2x on the FieldTyper scorer metric.

~25% cost reduction per run

Prompt caching cut cost by roughly 25% (about $6) per run.

Technology Utilized

Implementation Complexity

Risk & Compliance

Cost Reduction

Best Fit For

Best fit for product teams shipping LLM features who need rigorous, metric-driven evaluation to reduce hallucinations before scaling.

Chris Taylor

CEO & Co-Founder

Fractional AI

CEO & Co-Founder of Fractional AI, helping PE firms and portfolio companies implement AI workflow automations, product features, and diligence at scale.

Get an intro

Industry:

Technology & Software

Business Function:

Product & Engineering

Company Size:

Project Cost:

Cost not disclosed

Ownership:

Organization Type:

AI Pattern:

Document Processing & Extraction

Natural Language Processing

Value Type:

Risk & Compliance

Cost Reduction

AI Model:

Frequently Asked Questions

How did Zapier cut spec-generation hallucinations from 26% to under 1%?

By building a seven-metric LLM evaluation framework that measured correctness and hallucination rates across the spec-generation pipeline, then running iterative experiments. Targeted prompt engineering, passing extra context between stages, and model selection drove hallucinated endpoint paths from 26% down to under 1%.

What AI models and tools were used for Zapier's spec-generation evals?

The pipeline used GPT-4o by default and Claude Sonnet 3.5 for extraction, with evaluations run on the Braintrust platform against curated, manually reviewed OpenAPI specs and Postman collections. Prompt caching was added to reduce cost.

What results did Zapier achieve with the LLM evaluation framework?

Hallucinated endpoint paths fell from 26% to under 1%, field-type accuracy improved nearly 2x on the FieldTyper scorer metric, and prompt caching cut cost by roughly 25% (about $6) per run.

How long did the LLM evaluation project take?

A specific timeline was not disclosed. The work was iterative - building the metric suite, then running successive experiments on Braintrust and keeping only the changes that measurably improved accuracy.

Who is this LLM evaluation approach best for?

It is best suited to product teams shipping LLM features who need rigorous, metric-driven evaluation to reduce hallucinations before scaling.