Why Expedia Is Building AI Agents Through Evaluations, Not Requirements Documents

Encoding Intent Before Writing Code

At Expedia Group, product requirements documents no longer define what an AI agent should do. Instead, the travel platform encodes product intent directly into evaluation suites, a practice Xavi Amatriain describes as foundational to how the company ships intelligent systems. Amatriain, who joined as Expedia's first chief AI and data officer in December 2025 after leading AI and compute enablement at Google, frames evals as the canvas where teams articulate behavior, security constraints, and edge cases before a single line of agent code is written.

The logic is straightforward: if evaluation criteria capture what success looks like, including red-team scenarios and security thresholds, then those evals function as both specification and contract. In an environment where code itself is increasingly generated or assisted by AI tooling, Amatriain contends that human effort shifts upstream. The thinking goes into the evals; the implementation follows.

Yet the gap between aspiration and execution remains wide. Sixty-six percent of 157 enterprises surveyed already allow some production deployment of AI systems without human review or are building toward it within a year. Only 5% report full confidence in the automated evaluations that would govern those decisions. Half have shipped an agent that cleared internal checks but stumbled when a real customer interacted with it.

Three Layers Instead of Guardrails

Amatriain is skeptical of guardrails as a primary control mechanism. He calls them brittle, prone to interfering with the feedback loops that allow models to improve. When a system enforces artificial business rules or rigid constraints, it biases the user's behavior and distorts the signal the system learns from. The goal, he argues, is to minimize guardrails over time, even as he acknowledges they remain a necessary component in high-risk scenarios.

Expedia governs its AI stack through a three-layer architecture. Principles sit at the top, articulating how decisions should be made across a distributed organization. Amatriain is direct about the gap between principle and practice: most organizations, in his experience, fail to embed principles into culture, leaving them decorative rather than operational. The second layer consists of processes and tools that enforce those principles. Automation forms the third layer, executing checks and interventions calibrated to risk.

In practice, this manifests as agent release toll gates, checkpoints that scale governance intensity with the potential harm an agent might cause. Low-risk agents pass through lighter review; high-risk systems trigger red teaming, security audits, and multiple evaluation rounds. The structure is designed to be encoded and repeatable, shifting governance left into design rather than bolting it on after deployment.

Composing Specialized Agents, Not Monoliths

Amatriain's architecture philosophy rejects the idea of a single, unified artificial general intelligence. Even during his tenure at Google, he favored composition over monolithic models. Expedia's agentic systems are built from the component level up: tools compose into skills, skills assemble into sub-agents, and sub-agents are orchestrated into the full system.

This modularity serves both product and security objectives. Each agent is scoped narrowly, making it easier to evaluate, lock down, and reason about in isolation before it enters a larger orchestration. Unified principles define tone, context handling, and memory management across the system, but the design problem is systemic rather than model-centric.

The architecture also allows Expedia to match response strategy to latency requirements. A cached question about average hotel prices in Chicago during July can return instantly, drawing on retrieval-augmented generation. A query for a pet-friendly four-star property near Lake Michigan with a pool might justify a 30-second reasoning window, during which the system makes direct API calls to check real-time availability and cross-references supplier claims against traveler reviews.

That cross-referencing is central to Expedia's value proposition. A supplier might advertise a pool; reviews in Expedia's corpus might reveal it closes at 6 p.m. or was under maintenance during peak season. Generic chatbots surface self-reported data; Expedia's agents are designed to surface contradictions.

The User Keeps the Final Click

One design principle is non-negotiable: the agent does not book the flight or reserve the hotel. It recommends, discusses, and surfaces options, but the user must execute the transaction. Amatriain frames this as both a product and a security decision. Preserving user agency removes an entire class of failure modes and eliminates the need for a corresponding set of post-hoc guardrails.

Travel inventory is volatile. Pricing shifts in real time, seat maps change minute to minute, and availability windows close without warning. Allowing an agent to transact on behalf of the user introduces execution risk, liability, and the potential for costly errors. By keeping the final click with the human, Expedia constrains the blast radius of agent mistakes and maintains a clear boundary between assistance and autonomy.

Shifting Security Left, But Incidents Are Already Climbing

Amatriain advocates for security as a design principle, integrated as early as possible in the development cycle. Guardrails, in his view, are a symptom of security considerations arriving too late. When monitoring signals from production flow back into the eval suite, the feedback loop can approach full automation, allowing teams to detect, report, and fix issues rapidly.

But the margin for error is thin. Fifty-four percent of 107 enterprises surveyed in a separate June study reported an agent security incident or near-miss. Incident rates climb with organization size, reaching 63% among enterprises with more than 1,000 employees. Sandbox isolation, a control that limits damage after a breach, is adopted by just 20% of the largest organizations, down from 35% among smaller companies.

Amatriain warns that the threat landscape is shifting. Attackers will increasingly be other AI systems, probing for vulnerabilities with the same speed and scale that enterprises are deploying agents. Detection alone is insufficient; time to fix becomes the critical variable. The feedback loop from production signals to eval updates to patched code must tighten, or the adversarial cycle will outpace defensive iteration.

The Eval-First Playbook

Expedia's approach is a bet that encoding product intent, security requirements, and behavioral boundaries into evaluations before coding begins can shift the locus of control left. It assumes that automated evals, when designed with sufficient rigor, can replace both traditional PRDs and some portion of post-deployment guardrails. It assumes that composing specialized agents is more secure and more maintainable than training or fine-tuning monolithic models. And it assumes that preserving human agency at the transaction layer is a sufficient hedge against the autonomy risks that worry regulators and users alike.

The data suggests that bet is still being tested. Half of enterprises have already shipped agents that passed internal evals but failed in the field. Trust in automated evaluation remains low, even as deployment velocity accelerates. And incident rates are climbing faster than sandbox adoption, particularly at the scale where Expedia operates.

Whether evals can truly replace PRDs depends on whether teams can anticipate failure modes before they surface in production. The next year will reveal whether Expedia's three-layer governance model and agent release toll gates can keep pace with the adversarial systems Amatriain expects to emerge. For now, the feedback loop is live, and the evals are encoding lessons faster than any document ever could.