The first multi-agent system I built was at Google. At first glance, the product looked simple. A user opened a chat interface and asked a business question.
They could ask:
- →What was revenue last month?
- →Why did paid subscriptions decline?
- →Which customer segments contributed most to growth?
- →Which customers are likely to churn?
- →What will revenue be next quarter?
- →Can you build a churn prediction model?
The user did not need to upload a dataset, write SQL, identify a table, or select a machine-learning algorithm. The system was expected to determine what data was required, retrieve it from BigQuery, validate it, analyze it, and return an understandable answer.
But behind that simple interface was a complex, governed analytics workflow.
This was not just an exploratory data analysis tool. It was a conversational analytics and machine-learning system that combined natural-language understanding, metric governance, SQL generation, query validation, data-quality controls, analytical execution, machine learning, evaluation, and human oversight.
The Core Problem
Business users often know the question they want answered, but they do not always know:
- →which table contains the data;
- →how the metric is officially defined;
- →which joins are approved;
- →what filters must be applied;
- →which comparison period is appropriate;
- →whether the request requires descriptive analytics, diagnostic analysis, forecasting, or machine learning.
For example, a user may ask:
Why did paid subscriptions decline last month?
That question sounds straightforward, but the system must resolve several details before producing an answer.
A general-purpose language model could make assumptions about these questions. In an enterprise analytics environment, however, assumptions can produce incorrect business decisions.
The system therefore needed governance before generation.
The End-to-End Workflow
The final architecture followed this general flow:
Each stage had a clearly defined responsibility.
That separation became especially important after the first version exposed weaknesses in answer quality, hallucination, query reliability, and latency.
Intake Agent Versus Orchestrator
One of the most important design decisions was separating the intake agent from the orchestrator. They were not the same component.
The intake agent interpreted the user’s request. For a question such as “Why did paid subscriptions decline last month?” the intake agent extracted:
- →user intent;
- →primary metric;
- →requested date range;
- →comparison period;
- →dimensions;
- →filters;
- →business context;
- →potential ambiguity;
- →requested output type.
It also classified the task. Examples included:
- →“What was revenue last month?” — descriptive analytics
- →“Why did revenue decline?” — diagnostic analytics
- →“Which customers are likely to churn?” — predictive machine learning
- →“What will revenue be next quarter?” — forecasting
- →“Build a churn model.” — supervised model-development workflow
The output of the intake agent was structured rather than conversational. For example:
{
"intent": "diagnostic_analysis",
"metric": "paid_subscriptions",
"date_range": {
"current_period": "previous_calendar_month",
"comparison_period": "month_before_previous"
},
"dimensions": [
"country",
"acquisition_channel",
"subscription_plan"
],
"filters": [],
"requires_clarification": false
}The intake agent did not decide every downstream action. It understood and structured the request.
The orchestrator managed workflow execution. Its responsibilities included:
- →deciding which component ran next;
- →passing approved state between stages;
- →checking whether validation succeeded;
- →triggering retries;
- →stopping invalid workflows;
- →requesting user clarification;
- →enforcing human-review requirements;
- →preventing agents from bypassing controls.
We implemented the orchestrator primarily as deterministic workflow logic rather than as a completely autonomous AI agent. The AI could classify the request and recommend an analytical route. The application decided whether that route was permitted.
For example:
This distinction mattered because a free-thinking supervisor agent could skip steps, repeat work, create unnecessary model calls, or route a request directly into analysis before the data had been validated.
The orchestrator acted as the control plane.
Metric Governance and the Semantic Layer
Before generating SQL, the system retrieved approved business definitions. The language model was not allowed to invent the meaning of a metric.
For example, the approved definition of paid subscriptions could include:
- →official metric name;
- →metric description;
- →approved calculation;
- →required filters;
- →source tables;
- →approved joins;
- →time-column definitions;
- →dimension mappings;
- →data owner;
- →data freshness expectations;
- →access restrictions.
A structured metric definition drove SQL generation directly. For example, the generated query could look like this:
SELECT DATE_TRUNC(subscription_date, MONTH) AS subscription_month, country, acquisition_channel, COUNT(DISTINCT subscription_id) AS paid_subscriptions FROM `project.analytics.fact_subscriptions` WHERE subscription_status = 'active' AND is_trial = FALSE AND payment_status = 'paid' AND subscription_date BETWEEN @start_date AND @end_date GROUP BY 1, 2, 3 ORDER BY 1, 2, 3;
The SQL generation agent did not have permission to execute the query directly.
This was intentional.
The component creating the query was not allowed to be the only component deciding whether the query was correct or safe.
Three-Layer SQL Validation
We validated SQL using three complementary approaches:
- deterministic validation scripts;
- LLM-based semantic validation;
- human evaluation.
Each layer caught a different class of failure.
Deterministic scripts checked rules that could be enforced reliably in code. These checks included:
- →valid SQL syntax;
- →allowed statement types;
- →approved datasets and tables;
- →prohibited operations;
- →presence of date filters;
- →required metric filters;
- →approved join paths;
- →restricted columns;
- →use of query parameters;
- →row-level access requirements;
- →prohibition of SELECT *;
- →query-size limits;
- →estimated bytes processed;
- →timeout and cost thresholds.
The system also performed a BigQuery dry run before execution. The dry run helped estimate query cost and identify query errors without processing the full dataset.
Deterministic validation was appropriate for controls that should not depend on probabilistic reasoning. For example, an LLM should not be responsible for deciding whether a DELETE, UPDATE, or unapproved table reference is permitted. Those rules belong in code.
A query can be syntactically valid and still answer the wrong question. The LLM validation layer examined whether the generated SQL matched:
- →the user’s intent;
- →the approved metric definition;
- →the requested date range;
- →the comparison period;
- →the expected dimensions;
- →the analytical granularity.
For example, the LLM validator could detect that:
- →the query calculated total subscriptions rather than paid subscriptions;
- →trials were not excluded;
- →the date comparison used 30 days rather than calendar months;
- →the query grouped by country but omitted acquisition channel;
- →the query answered “what changed” but not “why it changed.”
The LLM validator returned a structured decision:
{
"status": "failed",
"issues": [
{
"type": "metric_mismatch",
"description": "The query does not exclude trial subscriptions."
},
{
"type": "date_mismatch",
"description": "The query uses rolling 30-day periods instead of calendar months."
}
],
"approved_for_execution": false
}Human reviewers evaluated SQL and answers against known business questions and expected results. The review process included:
- →predefined test questions;
- →approved reference queries;
- →known metric outputs;
- →edge cases;
- →ambiguous requests;
- →complex joins;
- →incomplete data scenarios;
- →questions that should trigger clarification.
Human review was especially important during early development because it exposed failure patterns that automated evaluations did not yet capture. For example, a response could contain technically correct SQL but still be misleading because the analytical interpretation overstated causality.
Human evaluation helped us assess the full user experience, not just SQL correctness.
Query Execution and Data Retrieval
After the SQL passed validation, the application executed the query through a controlled BigQuery service account.
The execution layer applied:
- →scoped credentials;
- →read-only permissions;
- →dataset allowlists;
- →timeouts;
- →scan limits;
- →row limits;
- →logging;
- →query identifiers;
- →user-level access checks.
Agents did not receive unrestricted warehouse credentials. They interacted with approved tools such as:
get_metric_definition() get_table_schema() validate_sql() estimate_query_cost() execute_bigquery_query()
This tool-based design reduced risk and made each action auditable.
Returned-Data Validation
Successful query execution did not mean the data was ready for analysis.
The returned dataset was validated before entering the EDA or modeling workflow.
- →whether any rows were returned;
- →required columns;
- →expected data types;
- →date coverage;
- →freshness;
- →missing values;
- →duplicate keys;
- →impossible values;
- →volume anomalies;
- →join duplication;
- →metric reconciliation;
- →granularity;
- →class balance;
- →target availability;
- →potential data leakage.
For example:
{
"status": "passed_with_warnings",
"row_count": 1864,
"date_coverage": {
"minimum": "2026-05-01",
"maximum": "2026-06-30"
},
"warnings": [
"Acquisition channel is missing for 4.8% of rows.",
"One regional source is 18 hours behind the expected refresh time."
],
"blocking_issues": [],
"approved_for_analysis": true
}This stage addressed an important reality:
A query can be valid while the returned data is still incomplete, stale, duplicated, or inappropriate for modeling.
Routing Into Analytics, EDA or Machine Learning
Once the dataset passed validation, the orchestrator routed it based on the original request.
What was revenue last month?
The system calculated:
- →totals;
- →averages;
- →counts;
- →rates;
- →trends;
- →period-over-period changes.
Why did subscriptions decline?
The system examined:
- →segment contributions;
- →country-level changes;
- →acquisition-channel changes;
- →plan-level changes;
- →shifts in customer mix;
- →statistically significant deviations;
- →correlated business drivers.
For an EDA request, the workflow evaluated:
- →distributions;
- →missingness;
- →duplicates;
- →outliers;
- →correlations;
- →cardinality;
- →target relationships;
- →temporal patterns;
- →class imbalance;
- →potential leakage.
For forecasting requests, the system validated:
- →time granularity;
- →historical coverage;
- →missing periods;
- →seasonality;
- →trend stability;
- →external-event considerations.
For supervised machine-learning requests, the workflow included:
- →target validation;
- →leakage checks;
- →train-validation-test splitting;
- →preprocessing;
- →baseline creation;
- →candidate model training;
- →cross-validation;
- →error analysis;
- →explainability;
- →model recommendation.
What Went Wrong in V1
Our first version demonstrated the potential of the system, but it also revealed several weaknesses. Only about 60% of answers consistently met our internal quality bar.
The most common issues included:
- →hallucinated metric definitions;
- →incorrect table selection;
- →invalid or incomplete SQL;
- →missing filters;
- →incorrect date interpretation;
- →unsupported causal claims;
- →analysis based on incomplete data;
- →repeated agent calls;
- →long response times;
- →inconsistent answer formatting.
The initial architecture gave too much freedom to the model. Some agents received overly broad prompts. Some responsibilities overlapped. Agents occasionally reinterpreted decisions that had already been made upstream. In some cases, the system generated SQL, executed it, and interpreted the results without enough independent validation between stages.
We also learned that adding more agents could increase latency without improving quality. Every additional agent introduced:
- →another model call;
- →another prompt;
- →additional state;
- →more potential failure points;
- →more opportunities for conflicting conclusions.
The goal therefore became controlled specialization, not maximum agent count.
How We Improved Answer Quality
We introduced several changes.
Agents stopped passing long unstructured narratives to one another. Instead, they exchanged typed objects containing:
- →intent;
- →metric;
- →dimensions;
- →filters;
- →date range;
- →query;
- →validation status;
- →warnings;
- →data profile;
- →analytical result.
The orchestrator enforced allowed workflow transitions. Agents could no longer bypass validation stages. The workflow knew exactly what should happen when:
- →a metric was unavailable;
- →SQL validation failed;
- →a query exceeded the cost limit;
- →data was incomplete;
- →user clarification was required;
- →human review was mandatory.
Metric definitions were retrieved from approved sources instead of being generated from model memory. This reduced inconsistent calculations and hallucinated business logic.
We separated SQL generation from SQL approval. Deterministic scripts handled security and rule-based checks. An LLM handled semantic alignment. Human evaluators reviewed representative queries and answers.
Returned data had to meet defined quality requirements before analysis could proceed. This prevented polished explanations from being generated from incomplete or unreliable data.
We created representative evaluation sets containing:
- →straightforward questions;
- →ambiguous questions;
- →questions requiring clarification;
- →complex metric calculations;
- →multi-table joins;
- →edge cases;
- →unsupported requests;
- →machine-learning requests.
We removed agents that did not provide meaningful independent value. Some tasks were converted into deterministic functions. Some tasks were performed in parallel. Others were skipped when the required result was already available in the shared state.
The system did not retry indefinitely. For example, a failed SQL query could receive one or two controlled revisions. After that, the workflow stopped or requested human intervention. This reduced latency and prevented expensive retry loops.
Structured state reduced interpretation drift between agents.
Results
After introducing stronger governance, validation, routing, and evaluation, answer quality improved from approximately 60% to roughly 85% against our internal quality criteria. We also materially reduced hallucinations.
Latency improved because the system made fewer unnecessary model calls, reused structured context, and moved deterministic work out of the LLM layer. The exact architecture was less important than the principle behind it: AI handled interpretation, planning, and explanation. Deterministic software handled access, policy enforcement, validation, execution, and workflow control. Humans remained responsible for evaluating critical outputs and approving high-impact use cases.
Key Trade-Offs
- →specialization;
- →clearer ownership;
- →better traceability;
- →easier testing;
- →better fault isolation.
- →higher cost;
- →greater latency;
- →more state-management complexity;
- →more failure points;
- →more conflicting outputs.
We learned not to create an agent for every small task. A component should become an agent only when it requires distinct reasoning, context, tools, or evaluation.
An AI supervisor can support flexible routing. However, fully autonomous routing is harder to test and govern. We chose a hybrid design. AI interpreted the user’s request. Deterministic code enforced the permitted workflow.
- →Does this SQL answer the user’s question?
- →Does the query align with the approved metric definition?
- →Is the explanation supported by the result?
- →Is the query read-only?
- →Does it reference an approved table?
- →Is a date filter present?
- →Does the query exceed the scan limit?
- →Is a restricted column included?
The strongest system used both.
Additional validation improved reliability but added processing time. We reduced the impact by:
- →running independent checks in parallel;
- →caching metric definitions and schemas;
- →using smaller models for classification;
- →limiting query revisions;
- →avoiding repeated context;
- →moving calculations into BigQuery and Python.
The system automated a significant portion of analytics work. It did not eliminate human responsibility. Human review remained important for:
- →metric-definition changes;
- →high-impact business decisions;
- →sensitive data use;
- →model-deployment approval;
- →causal interpretation;
- →new analytical workflows;
- →evaluation of unexpected outputs.
The Broader Lesson
A multi-agent system is not simply several prompts connected together.
A production system requires:
- →clear responsibilities;
- →governed context;
- →typed state;
- →deterministic orchestration;
- →restricted tools;
- →independent validation;
- →data-quality gates;
- →evaluation datasets;
- →observability;
- →human oversight.
The most important architecture decision was not how many agents we used. It was deciding which responsibilities belonged to AI and which belonged to traditional software.
- →understanding intent;
- →resolving language;
- →planning analysis;
- →interpreting results;
- →explaining findings.
- →authentication;
- →authorization;
- →SQL policy enforcement;
- →query execution;
- →data validation;
- →cost controls;
- →state transitions;
- →audit logging.
That separation helped move the system from an impressive prototype to a more reliable analytics product.
And that remains one of my biggest lessons from building multi-agent systems:
The intelligence does not come only from the model. It comes from the architecture surrounding the model.
About the Author. Tobe Awosanya is a Technical Founder and AI Product & Governance Leader working across AI engineering, agentic systems, AI evaluation, and technical AI governance. Her work focuses on translating AI product and governance requirements into practical system, control, and implementation decisions for teams building and adopting AI systems.
Considering Building a Multi-Agent System?
Multi-agent systems can look impressive in a demo and still fail in production because of weak orchestration, poor validation, unnecessary agent complexity, rising model costs, latency, and hallucinated outputs.
Through Data Techcon AI Advisory, I help founders and teams design practical multi-agent architectures that are easier to govern, evaluate, and scale.
- →determine whether your use case actually needs multiple agents;
- →define the right agent responsibilities and workflow;
- →separate AI reasoning from deterministic application logic;
- →design validation, guardrails, evaluation, and human-review controls;
- →choose the appropriate models, tools, data architecture, and orchestration framework;
- →identify cost, latency, security, and reliability risks before development begins.
Whether you are validating an early idea, reviewing an existing architecture, or preparing to move a prototype into production, you can book an advisory session to get expert guidance tailored to your product and use case.
Book an Advisory Session