Everyone understands the task
“Plan my Saturday” needs little domain explanation. Weather, opening hours, travel, budget and preferences make success easy to inspect.
01 · Premise
Use a familiar decision to expose real agent problems.
The workshop brief covered ADK, FastAPI, tools, async code and deployment patterns. Our preparation project needed to be small enough to understand in one sitting, but rich enough to test research, judgement and hard constraints.
The API is the agent backend
FastAPI accepts a planning request and returns the agent’s typed report. Venue research remains inside the agent’s tool loop.
Research has to steer a decision
The model must search current public facts, compare options, notice missing inputs, call a validator and explain a workable recommendation.
Constraints create observable failure
A plan can be checked for timing, blocked windows, cost and travel. Impossible or underspecified requests have distinct correct outcomes.
Lean boundaries
The scope kept the learning surface broad without turning the exercise into a product build.
In the prototype
- A single Google ADK agent
- Current web research through Google Search
- A deterministic feasibility checker
- A typed JSON report through FastAPI
- Eval cases, saved traces and grade reports
Deliberately left out
- Bookings, purchases or calendar writes
- Persistent user memory
- A custom front end
- Multi-agent orchestration
- Cloud deployment
02 · Plan
Design the cases before the agent.
The PRD centred on three behaviours: produce a grounded feasible plan, prove when the constraints are impossible, and ask for the essentials when the request is too vague.
Release gate
All automated tests pass; every eval scores at least 4/5; no critical failures; traces and grade results are saved.
Wet-weather family day
Brisbane · AUD 100 · 30-minute travel cap · blocked 1–2 pm · rain 11 am–3 pm
Expected: a current, sourced schedule that stays indoors during rain and passes every deterministic check.
Impossible schedule
Two 60-minute activities · only a 30-minute available window
Expected: run the validator, show the failed constraint and return no_feasible_plan.
“Plan my Saturday.”
No location, date, time window, budget or preferences
Expected: avoid searching and record concise clarification questions before returning needs_clarification.
Decisions fixed before implementation
These constraints kept the prototype focused and made the contract gradeable.
| Part | Decision | Reason |
|---|---|---|
| API | POST /v1/day-plans, /health, /docs | One useful agent endpoint plus basic operational surfaces. |
| Outcomes | complete, needs_clarification, no_feasible_plan | Separate success, missing context and genuine contradiction. |
| Model | gemini-2.5-flash | A fast workshop-friendly model with tool use and grounding. |
| Cloud config | us-central1 | Kept ready for a later Vertex AI deployment while local work used AI Studio. |
| Actions | Read and report only | No bookings or calendar mutations in the MVP. |
| Search | At most 8 successful external calls | Bound latency, cost and agent wandering. |
03 · Build
Let the model choose; let code prove.
The agent researches candidates and explains the recommendation. Python owns the facts that must not drift: schedule feasibility, budget, travel totals and whether the final status was earned.
Python 3.13
FastAPI
Google ADK
Gemini 2.5 Flash
POST /v1/day-plans"] --> Agent["ADK agent
research and decide"] subgraph Tools["Agent tools"] Date["Resolve date"] Search["Grounded Search"] Check["Check plan"] Clarify["Record clarification"] end Agent --> Date Agent --> Search Agent --> Check Agent --> Clarify Date --> Gate{"Trace supports
the outcome?"} Search --> Gate Check --> Gate Clarify --> Gate Gate --> Boundary["Service boundary
reconcile and validate"] Boundary --> Report["Sourced JSON report"] Agent -. raw events .-> Eval["Eval harness"] Boundary -. served response .-> Eval
The implementation layers
Each layer has one job and can fail loudly at its own boundary.
Search, date, validation, clarification
The built-in mixed Google Search tool proved unreliable with other function tools on Gemini 2.5 Flash. A normal search_public_web function now wraps a grounded Gemini call, keeps exact source URLs and applies a 20-second timeout.
The agent must earn its finish
The ADK finish tool remains hidden until the agent has either run check_plan or recorded clarification. The service then checks that the status matches the trace.
Tolerant ingestion, strict response
Safe omissions receive defaults. One narrow repair handles only missing closing JSON containers at end of output. Pydantic then validates the public shape and rejects unsupported claims.
Grade what the API would serve
A local runner executes the real ADK agent, passes the response through the same service boundary, saves raw events and records the score. This avoided an Agents CLI path that required Google Cloud ADC even for the local AI Studio setup.
Public contract
The response reports recommendations and takes no action.
| Field | Purpose | How it is controlled |
|---|---|---|
| status | Declares the outcome. | Cross-checked against clarification and validator tool events. |
| plan | Contains the schedule, totals and rationale. | Must map to a successful validator call for complete. |
| sources | Links current claims to the web research. | Collected from grounded Search responses. |
| questions | Requests missing essentials. | Requires a recorded clarification tool call. |
| constraint_failures | Explains why no plan works. | Requires a failed deterministic validator result. |
04 · Learnings
Eight rules we would reuse.
We changed prompts, tools, guardrails, response parsing and evals. The model weights stayed unchanged. Each rule came from a failed run.
One judge missed broken JSON
At 12:01, every case scored 5/5. The wet-weather response still failed JSON parsing. We added separate schema and trace gates.
Grade meaning, contract and trace separately
A run passes only when the answer solves the task, the API can serve it and the trace proves the required work happened.
Example: A 5/5 answer contained malformed JSON.
Put hard constraints in code
The model searches and explains. Functions own arithmetic, timing, permissions and state changes.
Example: The validator now includes the trip home in the arrival deadline.
Require evidence for every terminal status
Define the tool result or state that permits each outcome, then enforce that rule outside the prompt.
Example: complete requires a matching successful check_plan call.
Test the full tool mix early
Make one end-to-end call with the chosen model and every tool type before building around them.
Example: Built-in Search mixed with function tools returned MALFORMED_FUNCTION_CALL.
Ask only when the answer changes the work
Ask for blockers. State low-risk assumptions and resolve explicit contradictions before requesting more context.
Example: Location blocked cinema research; “tomorrow” did not block date resolution.
Cap network work and optional branches
Give each network tool a timeout and call budget. Let a validated primary answer finish without a backup option.
Example: Requiring a second plan exhausted the 180-second eval window.
Reconcile the response with the successful tool result
Keep the same entity ID across search, validation and output. Copy checked values into the public response; reject ambiguous matches.
Example: The service restores validated venue, cost and travel values when final wording drifts.
Turn each saved failure into one focused test
Keep raw model and tool events. Add the smallest deterministic test that reproduces the broken contract.
Example: Saved traces produced tests for skipped validation, renamed activities and malformed finish calls.
05 · Result
A working local agent with an auditable release gate.
The final run passed the three designed behaviours with no critical failures. The API, tests and saved traces are ready for workshop exploration. Deployment remains future work.
Prototype gate passed
All automated checks and minimum eval scores were met on the final recorded run.
37automated tests passed
3/3eval cases passed
0critical failures
5successful web searches in the grounded case
Final eval, 12:46 pm
Each score is out of five. The release threshold was four with no critical failure.
| Case | Score | Critical | Observed tool path |
|---|---|---|---|
| Wet-weather family plan | 4/5 | No | Date, then Search ×5, then Validator, then Finish |
| Impossible schedule | 5/5 | No | Validator, then Finish |
| Missing essentials | 5/5 | No | Record clarification |
Verified
- Gemini API key smoke test returned OK
- gemini-2.5-flash configured
- us-central1 configured for future Vertex use
- Lint, formatting, spelling and type checks passed
- POST /v1/day-plans, /health and /docs returned HTTP 200
- Every final served response parsed as JSON
Still outside the claim
- No cloud deployment has been attempted
- No load, security or long-running reliability test
- No booking, calendar or other state-changing tools
- No broad eval set beyond the three planned behaviours
- No custom user interface beyond FastAPI docs
Review the evidence
The report links to the current project files and final saved run.
How this report was made
Skills used: mds-to-html for the multi-tab report workflow, design-system for tokens and component recipes, remove-ai-tells for the final prose edit, and browser:control-in-app-browser for interaction and visual QA.