Why Zillow Measures AI Returns Before Deployment, Not After

The 40% Code Velocity Claim That Started Years Ago

When Zillow's engineering organization reports a 40% increase in code velocity tied to AI tooling, that number rests on a foundation the team started building long before any generative model entered production. Toby Roberts, the company's SVP of Engineering, recently explained that the measurement framework predates the AI push by years, a sequence that runs counter to how most enterprises approach AI attribution.

The company established DORA metrics across its engineering organization well before ChatGPT existed, tracking deployment frequency, lead time for changes, time to restore service and change failure rate as baseline signals. When the team later introduced AI-assisted coding tools, those baselines gave them clean before-and-after comparisons instead of anecdotal claims. Roberts emphasized that trying to retrofit measurement after deployment makes credible ROI nearly impossible to establish, especially when leadership asks whether the productivity lift is real or illusory.

At DailyTechWire, we've tracked similar patterns across Southeast Asian fintechs and e-commerce platforms over the past eighteen months. The companies that can articulate clear efficiency gains from AI deployments are almost always the ones that instrumented their workflows years earlier for reasons unrelated to machine learning. The ones scrambling to justify AI spend today are the ones trying to measure returns without any idea what normal looked like.

Context Proved Harder Than Data Infrastructure

Zillow's products touch roughly 80% of U.S. real estate transactions each year, according to the company, and a single customer journey can span months or years. A buyer might start on a mobile app researching neighborhoods, move to a phone call with a loan officer, then meet a real estate agent in person, and expect every interaction to pick up where the last one left off. No single chatbot interface can carry that thread across surfaces, timescales and human handoffs.

Roberts said the team identified early that they needed a persistent context layer that would meet customers and professionals wherever they showed up next, rather than forcing every interaction through a unified conversational interface. The engineering effort started where most enterprise AI projects start, with data foundations: a data mesh architecture, clear lineage and a governance structure that attached permissions and identity at the data layer itself.

None of that turned out to be the constraint. The hard problem was building something that remembered where a customer sat in their journey and carried that forward across disconnected touchpoints. The context layer had to live independently of any single model API or chat surface, a decision the team reached quickly once they mapped real transaction patterns instead of idealized conversation flows.

Small Models, Task-Specific Fine-Tuning and a Custom Harness

Zillow chose to build its own orchestration harness rather than route all interactions through a single frontier model API. The decision drew on two decades of machine learning history inside the company, including the Zestimate home valuation model that predates the current wave of generative AI by more than fifteen years. The architecture leans into smaller, task-specific fine-tuned models instead of relying on one general-purpose model to handle every use case.

That approach aligns with what we've observed across Asian markets, where latency, cost and data residency constraints push engineering teams toward smaller models and local inference whenever possible. Seoul-based game studios, Singapore wealth managers and Bangalore SaaS companies are all converging on similar patterns: use frontier models sparingly, fine-tune smaller models for repetitive tasks, and build the orchestration layer in-house to retain control over context and cost.

Internally, Zillow runs thousands of Glean agents in production alongside its custom harness, handling repetitive tasks across finance, legal and marketing with tens of thousands of executions. Glean's pitch, according to co-founder and CEO Arvind Jain, centers on centralizing integration work once through a gateway layer rather than letting every department rebuild connections to the same systems independently. That centralization also functions as a cost control mechanism.

Model Routing and Precomputed Context as Cost Levers

Jain outlined two mechanisms Glean uses to reduce token consumption and inference costs. The first is model routing, which directs most tasks to smaller, cheaper models instead of defaulting every request to a frontier model. The second is precomputed context, which avoids forcing an agent to assemble its own context from scratch on every invocation, a process that can burn tokens quickly and introduce significant latency.

Jain noted that assembling context dynamically can slow down even fast models, because the agent spends time and tokens reconstructing state before it can respond. Routing requests through a platform that maintains precomputed context can cut token consumption by as much as half, according to Glean's internal measurements. For enterprises running thousands of agent executions daily, that difference compounds quickly into material cost savings.

The cost lever argument matters more as enterprises move from pilot projects to production scale. Early AI deployments often ignore per-token costs because usage volumes are low, but once a system handles tens of thousands of daily invocations, inefficient context assembly becomes a line item finance teams notice. The companies that treat context as a cost optimization problem from the start, rather than an afterthought, are the ones that can scale without triggering budget alarms six months later.

Hard Rules Layered Over Permissions-Aware Architecture

Even with a permissions-aware context platform in place, Zillow layered additional hard rules and a standing compliance check on top for its most sensitive data categories. Roberts explained that the team chose not to trust the architecture alone to handle regulated data automatically, instead enforcing explicit constraints for financial and personal information.

That layering reflects a broader pattern we see across regulated industries in Asia. Banks in Singapore and Hong Kong, insurers in Seoul and healthcare platforms in Jakarta all follow similar strategies: build a strong permissions foundation at the data layer, then add explicit guardrails for categories where a permissions leak could trigger regulatory action or reputational damage. The architecture provides the baseline, but the business logic enforces the policy.

The decision also signals that Zillow treats AI governance as an active engineering problem rather than a configuration exercise. Trusting inheritance models and default behaviors works for many enterprise systems, but AI systems that assemble and generate responses dynamically introduce new failure modes. A misconfigured permission might expose a single document; a misconfigured context layer might synthesize and surface patterns across thousands of documents in ways that violate policy even when no individual access was improper.

Centralize Integration Once, Not Per Department

Jain's core argument for Glean's platform is that duplicated integration work across departments represents a hidden cost most enterprises haven't accounted for. Finance, legal and marketing teams often rebuild connections to the same internal systems independently, each solving the same authentication, rate limiting and error handling problems in parallel. Centralizing that work through a shared gateway eliminates redundant engineering effort and reduces surface area for security issues.

The argument maps cleanly onto the broader shift toward platform engineering that's been reshaping infrastructure teams across Asia over the past three years. Seoul-based e-commerce companies, Bangalore cloud-native startups and Singapore logistics platforms have all moved toward centralized developer platforms that abstract integration complexity once, rather than letting every product team reinvent the same wheels. Adding AI agents to that picture doesn't change the underlying economics; it just makes the cost of duplication more visible because agent integration work tends to be more complex than simple API calls.

For enterprises evaluating whether to build or buy their AI orchestration layer, Zillow's approach offers a useful data point. The company built its customer-facing context harness in-house because the shape of a real estate transaction didn't fit any off-the-shelf conversational interface, but it adopted Glean for internal agent deployments where standardization and centralized integration delivered more value than custom architecture.

What This Means for Enterprise AI Architecture

The session surfaced a few practical principles that cut across industries and geographies. First, measure before you build. Zillow's ability to attribute productivity gains to AI depends entirely on baselines established years before any generative model entered production. Enterprises that skip that step will struggle to justify continued investment once the novelty wears off and finance teams start asking hard questions about return.

Second, treat context as infrastructure, not a feature. The hardest problem Zillow faced wasn't data quality or model selection; it was building a layer that could carry context across disconnected surfaces and timescales. That layer required deliberate architecture and engineering investment, and it sits outside any single model API or vendor platform.

Third, centralize integration work once instead of letting every team rebuild it. The cost of duplicated effort compounds quickly as agent deployments scale, and the security surface area grows with every parallel implementation. Platforms that centralize that work, whether built in-house or adopted from vendors like Glean, pay for themselves in reduced engineering drag and lower operational risk.

Fourth, don't assume permission inheritance is enough for regulated data. Even with strong permissions at the data layer, enterprises operating in regulated industries need explicit guardrails and compliance checks for their most sensitive categories. AI systems that assemble and generate responses dynamically introduce failure modes that traditional access control models weren't designed to prevent.

Finally, cost optimization starts with architecture, not procurement. Model routing and precomputed context are both architectural decisions that reduce token consumption and inference latency. Enterprises that treat AI cost as a line item to negotiate with vendors, rather than a system design problem to solve with engineering, will find themselves constrained by costs that scale linearly with usage instead of leveling off as efficiency improves.

Jain's closing observation captures the shift enterprises are navigating right now: models by themselves don't bring automation inside the enterprise. Connecting them to enterprise context in a way that respects permissions, reduces cost and carries state across interactions requires deliberate platform work. The companies that recognize that early are the ones building systems that can scale beyond pilot projects into production workloads that finance teams will continue funding two years from now.