How we built our Reinforcement Learning Environment

1. From Pong and Breakout to a strategy laboratory

DeepMind's Atari work showed why reinforcement learning is more than scripting a clever move. An agent observes a game, chooses an action, receives a reward and updates its behavior from experience. In Pong, repeated interaction can improve paddle decisions. In Breakout, learned play can uncover useful patterns such as opening a route behind the bricks. The exciting part for us was strategy emerging from feedback instead of every useful sequence being written by hand. ^[1][2]

We borrowed that experimental idea, not the exact Atari implementation. DeepMind's early work used deep Q-learning and screen-based input; our BIOBUZZ work uses a multi-agent PPO approach and a modeled field. Neither the algorithm nor the observation interface should be treated as identical. The common principle is to make a game cheap to repeat, let policies explore, and judge the result against evidence. ^[2][3]

Our question is practical: which decisions help an alliance use its time and game pieces well? Collection, travel, shooting, HIVE access and parking compete for the same resources. A simulation lets us investigate those tradeoffs without wearing out hardware on every exploratory attempt. It narrows the questions for physical tests; it does not replace them. ^[1]

2. Start with the field, then make it behave

We began with field CAD and used Blender in the geometry and validation workflow. The strategy runtime became a headless, CAD-informed Python physics model with a 2D view for inspection. Geometry gives the environment a spatial foundation, but a recognizable field is not yet a trustworthy environment: pieces must move, collisions must matter and scoring must follow modeled state changes. ^[1]

The first prototype made that distinction obvious. It exposed idle behavior, incorrect field orientation, robots passing through structures, teleportation and missing GARDEN gathering. A high score in that version could reward an implementation mistake rather than a good decision. Correcting the world was therefore part of the learning project, not cosmetic work after training. ^[1]

The later release candidate described four equal-capability robots, 56 game pieces, 30 seconds of AUTO and 120 seconds of TELEOP actuation, with physical collisions, ball flight, HIVE tipping and parking represented. Those are configuration facts, not a claim of complete rule coverage. The official transition interval is not represented by those 150 powered seconds, and individual research studies can restrict the actions or objectives further. ^[1][4][5]

Two-dimensional strategy simulation also does not mean ignoring every airborne interaction. The model includes ball-flight behavior while replay presents the field from above. The development account does not provide enough equations or parameters to claim a particular integrator, collision algorithm or fully calibrated three-dimensional dynamics. ^[1]

3. Define the agent-environment contract

A learning environment needs a repeatable interaction loop: initialize a match, provide an observation, receive proposed actions, advance the model, compute feedback and identify when the episode ends. That is the conceptual contract we use to explain the system; it is not a published API specification. The source presentation does not enumerate the exact reset interface, tensor shapes, timestep or action encoding. ^[1]

The training account describes individually acting robots with limited local information and a coach that sees the full field during training. We distinguish an agent's observation from the trainer's broader state information. The presentation's local-vision explanation does not establish a particular camera, sensor model or deployment-ready perception stack. ^[1][3]

Moving, gathering, shooting, tipping and parking are the behaviors the environment seeks to represent. Physics and safety controllers sit beneath policy learning so that a proposed action cannot become a valid strategy simply by crossing a wall or teleporting. These modeled gates are not a complete competition referee or a physical safety certification. ^[1]

Rewards connect experience to the objective. The coach is described as evaluating how decisions help the team score, but the exact numerical reward function is not published in the presentation. Training reward, official scoring and evaluation criteria must not be conflated: a shaped reward is a learning signal, not automatically a competition point. We do not invent reward weights or claim that the current formulation eliminates reward exploitation. ^[1][3]

4. MAPPO searches for coordinated behavior

We use Multi-Agent Proximal Policy Optimization (MAPPO) to search for the best outcomes we can find under the modeled conditions. Multi-agent matters because one robot's choices change its partner's opportunities: collecting, reloading and approaching a HIVE are not independent scheduling problems. A policy that looks good alone may interfere with an alliance partner. ^[1][3]

MAPPO follows centralized training with decentralized execution. In the standard formulation, actors choose actions from their own observations while a centralized critic can use broader state information to estimate returns during training. The presentation calls that training role the coach. It should not be mistaken for a full-field controller secretly making every robot's decisions during execution. ^[1][3]

PPO-style updates use a clipped optimization objective to discourage overly large policy changes from one batch of experience. In plain language, learning should not abandon useful behavior because one unusual episode went well. Clipping is a stabilizing technique, not a guarantee that every update improves the policy. Network architecture, learning rates, clipping settings and training budgets are not specified here. ^[3]

The best outcome observed in a search is not necessarily an outcome learned by MAPPO. Our published full-match study makes this visible: all seven safety-qualified 13-TIP alliance outcomes came from scripted opponents, while the learning alliance reached at most 12. That distinction is essential. The environment demonstrated a modeled possibility; it did not demonstrate a learned policy that reliably achieves it. ^[4]

5. Monte Carlo tests the candidate, not the headline

After strategy discovery, we use Monte Carlo analysis to test candidate outcomes across repeated simulated conditions. The presentation describes a tester that varies lucky and unlucky factors and defensive scenarios, a challenger that supplies opposition, and a grader that evaluates unseen matches before promotion. MAPPO changes the policy from experience; Monte Carlo estimates how a chosen policy behaves under a specified sampling setup. Monte Carlo is not itself the policy-learning algorithm. ^[1]

One spectacular match is a witness of possibility. A distribution of matches answers a different question: how often does the behavior work, how variable is the result, and where does it fail? Mixing episodes from continually changing training policies can describe a search history, but it cannot by itself estimate the reliability of a single frozen policy. Our existing studies explicitly separate exploratory outcomes from frozen promotion evaluations. ^[4][6]

A stronger evaluation protocol freezes the candidate and incumbent, uses held-out seeds, compares them on matched conditions and balances alliance color or starting position where relevant. Useful reporting includes the full denominator, mean and spread, success rate for a stated target, and the frequency of geometry or safety failures. These are methodological recommendations, not claims that every development run used the same complete protocol. ^[6]

Safety filtering needs equal care. Report rejected matches as well as the retained population; otherwise a policy that often fails the constraints can look strong when only its surviving games are shown. If many candidates are selected on the same test set, selection can overfit that set, so a final held-out evaluation should remain separate. More samples reduce sampling uncertainty, but do not remove bias from an unrealistic environment. This methods paper reports no new confidence interval or new Monte Carlo run. ^[4][6]

6. Make experiments fast and outcomes inspectable

Throughput determines how many ideas we can afford to test. The development account describes a pivot toward headless physics and multiple worker processes rather than relying on Python threads for CPU-heavy collection. GPU/DirectML and AMD NPU inference were explored, but physics remained an important CPU bottleneck. Accelerator access alone does not make the whole simulation fast; the slowest substantial part of the loop still matters. ^[1]

We also changed what a replay stores. Early runs produced large MP4 recordings. Later work retained structured digital actor logs that could drive HTML Canvas playback. This separates match simulation from rendering: we can inspect recorded robot positions, game-piece motion and completion events without rerunning the match or treating a video as the only evidence. ^[1][4]

The embedded recording is an example of that evidence workflow. Playback starts paused, supports frame stepping and event jumps, and uses retained coordinates rather than inventing a new trajectory. It preserves the original state-based score, including 10 modeled field-content points per alliance at setup; those contents are not robot-earned actions or a fixed bonus. The linked study explains the source hashes and score interpretation. ^[4]

Selective recording has a tradeoff. Keeping detailed logs only for qualifying high-scoring matches makes the evidence archive smaller, but those highlights are not an unbiased sample. Retained witnesses explain a sequence; complete statistics and separately designed evaluations are needed to discuss reliability. Missing recordings should remain missing, not be reconstructed and presented as observed motion. ^[1][4][6]

7. Hill climbing improved the laboratory as well as the policy

Mustafa Suleyman's hill-climbing mindset is our inspiration for disciplined iteration, not a claim that he designed or endorsed this environment. Our working interpretation is simple: change something, measure what changed, retain the improvement and revisit failures. This is a development mindset; MAPPO's gradient-based policy optimization and the external promotion loop remain distinct mechanisms.

The development presentation traces nine iterations through CAD and Blender foundations, an early strategy prototype, mutation-based search, physics corrections, MAPPO, throughput experiments, replay redesign, score-bound studies and a source-pinned study series. The progression was not just toward larger numbers. It was toward stronger reasons to trust or reject a number. ^[1]

That gives the project two coupled loops. The inner loop learns policies inside a version of the environment. The outer loop examines physics, constraints, evaluation and evidence, then decides what to improve or promote. A better-looking policy score may be caused by a model change, so versions and assumptions must travel with the result. Preserved studies should remain tied to their original evidence rather than silently acquiring a new interpretation. ^[1][4][6]

8. What the RLE gives us, and what comes next

The RLE gives Cookie Coders a repeatable place to ask strategy questions and a way to inspect why a sequence succeeded. It can reveal coordination opportunities, expose simulator defects and prioritize physical experiments. It cannot turn a modeled score into a robot record, establish a global optimum, or remove the need to measure hardware. ^[1][4]

Important assumptions remain. The presentation describes collection in all directions and scoring from positions with line of sight; the published studies also name idealized sensors, equal hardware and uncalibrated shot and HIVE-motion parameters. Those assumptions can make a strategy easier in simulation than on a real robot. Calibration needs measured movement, collection, shooting, HIVE dynamics and representative opposition before transfer claims become credible. ^[1][4][6]

A reproducible implementation paper should additionally pin code and dependency versions, observation and action definitions, the numerical reward function, physics parameters, MAPPO configuration, seeds, evaluation splits and promotion criteria. The presentation supplies a development account, not that full reproduction package. Students own the next questions and explanations; physical robot experiments remain adult-supervised. Our conclusion is deliberately modest: learn in simulation, test across variation, inspect the evidence, then measure on the robot.

The build, separated by responsibility

A conceptual architecture summary grounded in the development presentation, not a benchmark or an executable interface specification.

Source [1].