Cisco AI Introduces FAPO: Pipeline-Aware Prompt Optimization With Step-Level Failure Attribution and Claude Code Orchestration

Getting prompts right is still the hardest part of shipping reliable LLM applications. Small wording changes can swing accuracy by 20 percent. What works on a few examples often breaks at scale. When a multi-step pipeline returns a wrong answer, finding the failing step means inspecting intermediate outputs by hand.

Cisco AI introduced FAPO to address that bottleneck. FAPO stands for Fully Automated Prompt Optimization. It is a Claude Code-driven system that optimizes LLM pipelines from baseline prompts to target accuracy. You supply a dataset and an initial prompt. FAPO then evaluates, classifies failures, proposes variants, validates them, and iterates. The whole loop is orchestrated by Claude Code agents. The project ships open source under Apache 2.0, and also supports Codex as the optimization agent.

In Cisco’s reported evaluation, FAPO beat GEPA, a state-of-the-art prompt optimizer, on 15 of 18 model-benchmark comparisons. On the two benchmarks where FAPO escalated to pipeline changes, the mean gain over GEPA reached +33.8pp.

TL;DR

• FAPO is a Claude Code-driven system that autonomously optimizes multi-step LLM pipelines from baseline prompts to target accuracy, open source under Apache 2.0.
• It escalates through three levels — prompt, parameter, then chain structure — using step-level failure attribution to decide what to change next.
• In Cisco’s evaluation, FAPO beat GEPA on 15 of 18 model-benchmark comparisons, with a +14.1pp mean gain.
• On HoVer and IFBench, where it escalated to pipeline changes, FAPO won all six pairs at a +33.8pp mean gain; AIME was GEPA’s only win, within sampling noise.
• Guardrails against overfitting include training-split-only inspection, immutable variant files, and an independent reviewer on every proposal.

What is FAPO

FAPO is a multi-tenant evaluation and optimization framework. A tenant is a self-contained optimization project. Each tenant directory holds one task’s prompts, dataset, chain definition, scorer, and config. Tenants stay isolated, so unrelated tasks optimize side by side without interference.

The core engine is named hephaestus and is domain-agnostic. It handles evaluation, chain execution, and scoring. Chains are LangGraph state graphs that process each test case. Out of the box, FAPO supports three providers: OpenAI, Baseten, and SageMaker.

The one input you must bring is a dataset. It is paired inputs and expected outputs that define success. FAPO splits it into a validation set and a held-out test set. The validation set drives iteration; the test set is used only for a final one-shot evaluation. From a task description, Claude can scaffold the rest: the initial prompt, the chain, and the scorer.

How the Optimization Loop Works

Once the pieces exist, FAPO runs a closed loop until target accuracy is reached. Each cycle runs six stages:

1. Evaluate — run the chain on the dataset, collect per-case scores and step-level outputs.
2. Attribute — classify failures by root cause using rule-based heuristics plus LLM analysis.
3. Propose — generate a variant targeting the dominant failure cluster.
4. Review — an independent agent validates the proposal for scope compliance and data leakage.
5. Compare — accept the variant only if it improves on the previous best, otherwise reject.
6. Iterate — continue until target accuracy is reached or the optimization budget is exhausted.

The system works at three escalating levels. Prompt edits are lowest cost and tried first. Parameter changes adjust config values like retrieval_k or temperature. Structural changes alter chain topology, such as adding a self-reflection node or switching to a ReAct pattern. FAPO exhausts one level before escalating to the next.

Step attribution sorts failures into four classes. Retrieval failures return empty or irrelevant content. Cascading failures begin when an early step produces empty output. Format failures hide the correct answer inside text the scorer cannot parse. Reasoning failures occur when good inputs still produce a wrong conclusion. Format and reasoning issues are prompt-addressable. Retrieval and cascade issues are structural-addressable.

Guardrails keep the optimizer from overfitting. It inspects only training-split cases, while validation and test expose aggregate scores only. Every variant is a new immutable file, never edited in place. An independent reviewer checks each proposal before it runs.

The Benchmark Case: FAPO vs. GEPA

Cisco team evaluated FAPO against GEPA (Generalized Evolutionary Prompt Architecture), a state-of-the-art prompt optimization method. GEPA uses evolutionary search with genetic operators to optimize prompts for multi-step pipelines. Both systems started from identical baseline pipelines and prompts. FAPO could escalate to structural changes when attribution found bottlenecks. GEPA was limited to prompt-level optimization.

The comparison spanned six benchmarks and three task models: GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B. Claude Opus 4.6 served as both FAPO’s orchestrator and GEPA’s reflector. Scores below are averaged across the three task models.

FAPO won 15 of 18 model-benchmark comparisons, with a mean gain of +14.1pp over GEPA. On HoVer and IFBench, where FAPO escalated to pipeline changes, it won all six model-benchmark pairs. The mean gain there was +33.8pp. On the four benchmarks without structural changes, FAPO still won 9 of 12 through prompt optimization alone. AIME was the only benchmark where GEPA led, by 3.1pp. The gap is smaller than the standard deviation across stochastic trials.

A capability comparison shows the design difference reported by Cisco. Every row below reflects the source description of the two systems.

Where It Fits: Use Cases

FAPO targets multi-step LLM pipelines, not single prompts. A few concrete examples:

• Multi-hop question answering: A chain retrieves documents, extracts facts, reasons over evidence, and formats an answer. In Cisco’s documented walkthrough, a multi-hop QA chain rose from 39.3% to 70.3% validation exact match across two iterations. Attribution then flagged the remaining failures as retrieval-limited, signaling a structural fix. Separately, on the HotpotQA benchmark, FAPO reached 68.3% test accuracy versus GEPA’s 61.8%.
• Instruction following: On IFBench, format-constraint failures pushed FAPO to escalate beyond prompts, reaching 80.7% test accuracy.
• Classification: A software-name-to-category task can be scaffolded by Claude Code, then optimized to exact-match targets.
• ReAct agents: An MCP workflow extension optimizes a tool-calling ReAct agent using trajectory scoring and LLM-as-Judge scoring.

Getting Started

The fastest path is to let Claude Code create the tenant files. From the repo, describe your task in plain English, then add a JSONL dataset. Each line is one test case with case_id, task_type, context, expected, and metadata:

{"case_id": "1", "task_type": "qa", "context": {"question": "What is the capital of France?"}, "expected": {"answer": "Paris"}, "metadata": {}}
{"case_id": "2", "task_type": "qa", "context": {"question": "What is 2 + 2?"}, "expected": {"answer": "4"}, "metadata": {}}

A scorer compares the chain output to the expected answer. It implements validate_case to catch bad data early and score_case to return a composite score:

from hephaestus.scoring.scorer import Scorer as BaseScorer

class Scorer(BaseScorer):
    def validate_case(self, case, scoring_profile):
        assert "answer" in case.expected, "Missing 'answer' in expected"

    def score_case(self, case, output_text, scoring_profile):
        expected = case.expected["answer"].strip().lower()
        predicted = output_text.strip().lower()
        em = 100.0 if predicted == expected else 0.0
        return {"composite_score": em, "score_breakdown": {"exact_match": em}}

Verify the setup with a baseline evaluation:

export OPENAI_API_KEY="sk-..."
python -m hephaestus.cli eval --config tenants/my_project/configs/eval.json

Then invoke the optimization agent with a tenant, config, and success criteria such as composite_score >= 90. Claude Code produces a scope contract, then iterates autonomously. Every prompt variant, config, and per-variant analysis is written to disk, so each run stays auditable. A local read-only UI called FAPO Explorer browses the artifacts afterward.

Strengths and Weaknesses

Strengths

• Pipeline-aware scoring attributes failures to the step that caused them, not just the final output.
• Three-level escalation handles failures that prompts alone cannot fix.
• Guardrails against overfitting: training-split-only inspection, immutable variants, and an independent reviewer.
• Open source under Apache 2.0, with both Claude Code and Codex supported.

Weaknesses

• Optimization quality is bounded by the dataset’s quality and coverage, which you must supply.
• The project is recent, so independent production track records are still limited.
• The default loop depends on agentic coding tools (Claude Code or Codex) rather than a standalone optimizer.

Interactive Explainer