Training Multi-Turn Agentic LLMs with Tunix on Google TPUs

Training Multi-Turn Agentic LLMs with Tunix on Google TPUs

Finer per-turn credit assignment remains a critical area for improvement in reinforcement learning agents that currently rely on single trajectory-level reward systems. As the technology landscape of 2026 demands more autonomous and reliable AI systems, the shift from static text generation to interactive problem-solving has become the new industry standard. Google Tunix provides a streamlined framework for supervised fine-tuning and single-turn Group Relative Policy Optimization on Cloud TPUs, allowing developers to manage the complexities of modern agentic workflows. By utilizing Cloud TPU v6e-8, organizations can now handle the massive computational requirements of multi-step reasoning models without sacrificing efficiency or accuracy. These models must not only generate text but also observe environmental changes, invoke external tools, and refine their internal strategies based on feedback received over several conversational turns. This transition toward agentic behavior represents a significant leap in how Large Language Models are integrated into practical, real-world applications where a single response is rarely sufficient to complete a complex task. The integration of advanced policy optimization techniques within the Tunix ecosystem ensures that these agents can be trained to handle the nuances of multi-turn interactions, maintaining consistency and logic across extended sequences of actions and observations in a dynamic environment.

1. Establishing the Strategic Plane Strike Environment

The training process begins with a specialized board game known as Plane Strike, which serves as a robust testbed for spatial reasoning and multi-turn decision-making. In this scenario, the agent operates on a 6×6 grid, ranging from coordinates A1 to F6, where it must locate and hit an eight-cell plane hidden in one of four orientations. Unlike simple classification tasks, this environment requires the model to interpret visual board states represented by untried cells, hits, and misses. The model, specifically Gemma 4 E2B in its native thinking mode, must evaluate the results of its previous actions to deduce the hidden geometry of the plane. Each turn represents a cycle of observation, reasoning, and tool execution, making it an ideal candidate for reinforcement learning. The agent is restricted to a maximum of 23 strikes per game, forcing it to develop efficient search patterns rather than relying on brute-force exploration. This setup mirrors many real-world diagnostic or search tasks where information is revealed incrementally and resources are constrained by a strict operational limit.

Initial benchmarks indicate that standard models, even those with significant reasoning capabilities, struggle with the spatial complexities of Plane Strike without targeted training. Out of the box, the base model typically achieves a win rate of only 9.4% on held-out game boards, averaging over 22 strikes per session. This low success rate highlights the difficulty of translating raw reasoning power into a coherent tactical strategy within a constrained multi-turn environment. The objective of using agentic reinforcement learning is to transform this baseline performance through iterative policy updates. By applying 100 steps of agentic Group Relative Policy Optimization via the Tunix pipeline, the target is to elevate the win rate to 95% while simultaneously reducing the average number of strikes required to approximately 16. This significant improvement demonstrates the power of reinforcement learning in teaching models to recognize patterns and optimize their tool-calling sequences based on the rewards generated by successful hits on the board.

2. Architecting the Modular Agentic Pipeline

Constructing an effective multi-turn reinforcement learning pipeline requires a modular approach to ensure scalability and ease of debugging across the TPU infrastructure. The Tunix framework organizes this pipeline into four primary components: the task environment, the tool and chat template parser, the conversation agent, and the learner responsible for trajectory collection. This separation of concerns allows developers to modify the environmental logic or the tool-calling schemas without disrupting the core reinforcement learning algorithm. In the context of 2026, where model architectures and hardware capabilities evolve rapidly, this modularity is essential for maintaining a flexible research and development workflow. Each component is designed to interact through a series of standardized interfaces, ensuring that data flows seamlessly from the environmental feedback loop into the policy optimization stage. This architecture supports the intensive data requirements of agentic training by distributing the workload across multiple TPU chips, maximizing throughput during the complex rollout phase.

The environment logic is encapsulated in a specialized file that subclasses the standard task environment to implement the specific rules of the board game. Each environment instance is initialized with a specific seed that dictates the hidden position of the plane, ensuring that all rollouts within a specific group encounter the exact same board configuration. This consistency is critical for the reinforcement learning algorithm to accurately compare different strategies and assign credit to the most effective actions. As the game progresses, the environment tracks every strike, evaluates the outcome against the hidden board, and updates the visible grid for the next turn. It also calculates the step-wise rewards that are eventually aggregated into a trajectory-level reward for the entire episode. By managing the state of the game board and providing structured feedback to the agent, the environment serves as the ground truth against which the model’s performance is measured and improved throughout the training cycle.

3. Implementing Tooling and Conversational Logic

Effective tool integration is the cornerstone of agentic behavior, and it requires a sophisticated parsing system to handle the transition between reasoning and action. Using a custom chat template parser, the system appends tool declarations directly to the system prompt, informing the model of the available strike functions and their required parameters. Gemma 4 utilizes specific channel tokens to separate internal thought processes from external tool calls, ensuring that the model’s reasoning remains transparent and structured. When the model determines a coordinate to strike, it generates a specific tool call sequence that the parser must identify and translate into an environmental action. This process is supported by native stop tokens that signal the model to pause its generation and wait for a response from the environment. This interactive loop ensures that the model does not attempt to predict its own outcomes but instead reacts to the authentic data provided by the game environment after each move is executed and processed.

Managing the flow of a multi-turn conversation involves tracking the history of observations, thoughts, and responses to maintain a coherent state over time. The conversation agent acts as the intermediary, translating raw environmental observations into a format that the model can interpret, such as a textual representation of the current board. If the model produces an invalid action, such as striking a cell that has already been hit or providing a coordinate outside the grid, the agent must handle these errors by providing corrective feedback or assigning a penalty within the reinforcement learning framework. This robust handling of edge cases is vital for training an agent that is not only smart but also compliant with the operational rules of its environment. By maintaining a detailed trajectory of all steps taken during a game, the agent provides the necessary data for the learner to analyze which sequences of reasoning and action led to a successful outcome, further refining the policy for future iterations.

4. Scaling Training with TPU Rollout Engines

The computational demands of multi-turn reinforcement learning are addressed by integrating high-performance rollout engines that leverage the unique architecture of Cloud TPUs. To prevent bottlenecks that often occur during variable-length rollout generation, the pipeline utilizes an asynchronous generation mode via specialized TPU-optimized backends. This allows the system to continue generating new trajectories while simultaneously performing policy updates, maximizing the utilization of the TPU v6e-8 host. Prefix caching is also enabled to store the growing conversation history of each game, significantly reducing the redundant computation required as the agent reaches later turns in a multi-turn session. Since the actor model, the reference model, and the rollout workers share the same hardware host, efficient memory management is paramount. This configuration ensures that the high bandwidth and low latency of the TPU interconnect are fully utilized to maintain a high throughput of training data even as the complexity of the agentic tasks increases.

Configuring the learner involves setting specific parameters for policy optimization to ensure stability and convergence over the course of the training job. The system uses a token-based loss algorithm and a specialized advantage estimator to preserve the continuity of token identifiers across multi-turn rollouts, which prevents the retokenization drift that can sometimes plague complex RL workflows. The learning rate is carefully scheduled to allow for rapid initial progress followed by a gradual refinement of the policy as the agent approaches near-optimal performance. By integrating the actor and reference models within a unified engine, the system can efficiently calculate the relative improvements of the current policy against a stable baseline. This setup allows the training process to scale across the available TPU chips, providing the necessary processing power to handle hundreds of thousands of turns across thousands of game boards in a relatively short amount of time, proving the efficiency of the Tunix-on-TPU approach in 2026.

5. Analyzing Performance Metrics and Strategic Growth

Upon completion of the training process, a detailed analysis of the performance metrics reveals significant advancements in both the agent’s logic and its operational reliability. One of the primary indicators of success is the tool-calling compliance rate, which typically reaches near-perfect levels early in the training cycle. The frequency of invalid actions, such as improperly formatted commands or redundant moves, drops to a negligible level as the model learns to adhere strictly to the environment’s constraints. This foundational instruction-following capability is essential for the model to then focus on the more complex task of strategic deduction. As the training progresses, the win rate on evaluation boards climbs dramatically, indicating that the agent has successfully moved beyond random exploration and has developed an internal model of the game’s mechanics. These improvements are supported by rigorous statistical analysis of the training trajectories and the distribution of rewards across various game scenarios.

The efficiency of the agent is further demonstrated by the substantial reduction in the average number of strikes required to finish a game. Reducing the strike count from an average of 22 down to 16 proves that the model is making more informed decisions and is effectively utilizing the feedback from hits and misses to narrow down the possible locations of the plane. This strategic growth is a direct result of the agentic reinforcement learning process, which penalizes unnecessary moves and rewards the rapid completion of the task. While the current implementation relies on trajectory-level rewards, these results suggest that even more granular performance could be achieved in the future by implementing finer per-turn credit assignment. The success of this project serves as a proof of concept for deploying similar agentic systems in other domains that require multi-turn reasoning and tool integration, such as software debugging, automated research, or complex customer support interactions where the path to a solution is not immediately evident from the initial prompt.

The successful implementation of agentic reinforcement learning with Tunix demonstrated that complex decision-making could be mastered through structured multi-turn training. By integrating Gemma 4 with specialized TPU hardware, the project established a blueprint for future developments in autonomous AI systems. The transition from static text generation to dynamic environmental interaction proved that high-performance infrastructure was the primary driver of agentic efficiency. Future projects should consider exploring diverse reward structures to further refine the reasoning capabilities of multi-turn models. Implementing granular credit assignment models remained the most promising path for enhancing deduction in environments with high sparsity. These findings provided a clear roadmap for organizations aiming to deploy sophisticated agents in real-world scenarios requiring persistent logic and tool integration. It was evident that the combination of modular software frameworks and high-bandwidth memory architectures would continue to define the boundaries of what large language models could achieve in 2026 and beyond. As the landscape evolved, the emphasis shifted toward creating agents that not only responded but anticipated the needs of the task at hand through continuous learning and environmental adaptation.

Subscribe to our weekly news digest.

Join now and become a part of our fast-growing community.

Invalid Email Address
Thanks for Subscribing!
We'll be sending you our best soon!
Something went wrong, please try again later