Developing with AI agents: from scoping to tests on real devices

Prepare agent work and the tools to test it

My enterprise full-stack experience supports scoping, architecture choices and integration. With Codex and Claude Code, I also prepare the working context and tools that let agents develop, interact with the system and check their changes. I retain product decisions and responsibility for accepting a result.

MultiPOV, Aidventures and Nomad show how I adapt that preparation: access to an Android WebView, a CLI sharing the game engine, or tasks split between Rust and Flutter specialists based on hands-on feedback.

Scope through voice, then hand off the decisions

I start in regular ChatGPT chat, using voice dictation. I describe the need, compare approaches and clarify constraints. I decide on scope, priorities, development order and acceptance criteria. This also covers the tools and access needed: how the agent will access the system, which actions it can perform and which tools it can use to test and validate.

Once these choices stabilise, I request specifications and handoff instructions for the agent working in the repository. They must retain the decisions and their reasons, constraints, criteria and required access. If I choose to address the complex core before the interface, that priority must reach the development task.

For each handoff, I also request a check of relevant, current official OpenAI guidance to adapt the instructions to the tools involved.

Scope the need and the means with a conversational agent → hand off specifications and access → prepare Codex or Claude Code tasks and tools → execute, observe, diagnose, correct and retest.

Prepare instructions, roles and checks

In the repository, I prepare instructions and reusable skills, roles and bounded tasks. A task must state what the agent should change, the useful context, constraints and expected checks. I also specify the version to test, the scenario to run, the signals to observe and the failure criteria.

When tasks are independent, I can split exploration, implementation and review between distinct roles. I combine their findings and resolve disagreements. Reviews cover the code as well as the environment and reliability of testing tools: a check supports a decision only if we know what it actually observed.

MultiPOV: implement and test at the right layer

MultiPOV synchronises several video viewpoints. Its evolution towards Rust/WASM and Android is covered in the media-system article.

To work on an Android problem, I wanted the agent to reproduce the journey on the phone and identify the build involved. The MCP connected through CDP lets it drive the WebView: read the DOM, use buttons and check the control room’s state. ADB complements this access with identification of the installed APK, native logs, screenshots and screen recordings.

The working loop is: implement → test the WebView through MCP/CDP on an identified build → consult native logs through ADB when needed → diagnose and correct → retest. A correct DOM state provides information about the web interface; it does not prove that native Media3/ExoPlayer video renders correctly. Checking what the user sees requires observations suited to that layer.

SyncBench provides a harness for audio/video observations and explicit criteria. I define scenarios and thresholds, then assess whether the observations are sufficient to accept the result. Qualification remains tied to the scenario and version: VOD, live, web and Android need distinct checks.

Correct the test harness too

A harness review exposed a possible false success: when measurement could not observe the content, it could confuse that failure with actually blocked content. The verdict must then be inconclusive, with the missing observation reported. For synchronisation, I also wanted to account for the time offset so that an apparently acceptable frame count would not conceal an excessive duration.

An agent can examine the harness rules and propose a fix. I retain responsibility for the phenomenon to measure and the acceptance criteria. This case requires reviewing the testing tool before trusting its verdict.

Aidventures: work on the core first through a CLI

For Aidventures, I chose to address the complex game core first and make it usable from the command line. The CLI shares the engine, game database, lore, model providers and orchestrator with the web application. Codex and Claude Code can therefore run a game scenario, resume a session and examine a state transition without depending on the graphical interface.

The loop is: implement in the shared core → run a CLI scenario → analyse engine traces → correct → replay and check regressions. Replay rereads a trace without another API call. Simulations and replays therefore do not validate calls to a real provider. The AI game master belongs to the product; the agents developing the application have a different role.

Retrieve LLM calls and state transformations

In instrumented audit scenarios, I want to retain the request sent to the LLM and its raw response text, before parsing and drift-guard corrections. These captures retain messages, the model, generation parameters, responses and call errors. Engine traces track drift checks and changes to game state.

If an item is missing from the final inventory, an agent can compare what the model proposed, what the drift guard accepted or corrected and what the engine applied. It can then look for the discrepancy in the prepared context, provider call, response, parsing or drift-guard processing.

Inventory mismatches encountered during playtests required a second review to distinguish a trace-analysis problem from a context-sharing defect. Diagnosis relied on actions and state changes, followed by regression tests.

The environment also needs checks: an audio service stopped responding after files moved. Diagnosis identified an invalid launch path. Its correction was followed by an audio-stream retest and the addition of a path check to CI. For this problem, verification used the interface and running services.

Nomad: start with the need and hands-on feedback

Nomad addresses my need to continue developing from a phone, using voice to dictate requests and a Joy-Con to control the remote machine. The journey covers the connection to Linux, input into development tools and restoration of the usual environment after disconnection.

I organised work with Claude Code around Rust, Flutter, patch maintenance and distribution roles. Each role receives the context and constraints for its part. The RustDesk foundation requires accounting for the original project when making adaptations; I retain integration decisions.

The loop is: need or hands-on feedback → context and task for the relevant role → implementation → appropriate checks → correction and retest → integration review. For a Rust change, those checks include Rust tests and Clippy. Hands-on feedback remains necessary to assess the mobile journey.

Disconnection could, for example, return the screen to a fixed orientation. I wanted to restore the state that existed before the session. This feedback led to a fix and regression tests. Integration review must then check that the behaviour fits the complete journey.

Accept results based on observations

I retain product decisions, context preparation, task boundaries and acceptance of results. Agents can implement, run journeys they can access, diagnose and retest; I must also decide whether the environment and tools allowed sufficient verification.

A result must identify the scenario, version and observations: passed when the criteria are met, failed when a discrepancy is observed, inconclusive when observations do not allow a decision, and not applicable when an action falls outside the tested context.