The early version of this rig kept walking into walls. Seven depth rays fanned out across the player’s field of view, and a doorway narrow enough to slip between two of them simply did not exist as far as the AI was concerned. It would shuffle forward, hit drywall, back up, and try the same wall again. Andrea Ricci pushed the scan to 51 rays, and the corridors of Doom finally opened up.
What Ricci actually built
The project pairs a SCINTIX P4 system-on-module with Anthropic’s Claude Sonnet and hands the model the controls of Doom on real hardware. The SoM is built around Espressif’s ESP32-P4, with an ESP32-C6 riding alongside for wireless. Doom runs on the board itself rather than streaming from a PC, drawing to a 7-inch 1024×600 MIPI-DSI panel. The game still renders internally at its original 320×200, so the ESP32-P4’s Pixel Processing Accelerator does the upscale in hardware. That single offload cut frame scaling from roughly 24 ms to about 9 ms and held the game at a steady 30 FPS.
How the AI sees the level
The clever restraint here is what Claude is not allowed to know. Instead of reading the game’s internal state, the firmware exposes Doom over a lockstep WebSocket. The model issues one action, move, turn, fire, open a door, or swap weapons, the engine advances a handful of frames, then freezes and hands back a fresh observation. That observation is deliberately thin: the 51-ray depth scan for wall distances and openings, a list of enemies and items visible on screen, an ASCII automap covering only the walls already explored, and two hints flagging a closed door ahead or a move that got nowhere. Monsters behind walls stay invisible, the same way they would for you.
Getting there was the usual embedded slog. Past the seven-ray blindness, the real gotcha was sign convention. Reported bearing angles ran opposite to the documentation, so Claude turned left with total confidence when it meant right, right up until every component agreed on one coordinate system.
The takeaway
Claude is a mediocre Doom player. It stalls a second or two every few frames while it works out its next move, and it fights like someone reading the manual mid-firefight. That is fine, because the kill count was never the interesting part. The engineering worth stealing is the perception layer and the lockstep loop, and both port down to cheaper silicon than an ESP32-P4. If you want to try the pattern on hardware you already own, start small: expose a handful of state values over a WebSocket from a plain ESP32 using any of the common Arduino WebSocket libraries, then drive it from a script on your laptop before a model touches it. Ricci’s full write-up is on Hackster.
Frequently Asked Questions
What hardware does this Doom build run on?
A SCINTIX P4 system-on-module built around Espressif’s ESP32-P4, paired with an ESP32-C6 for wireless. Doom runs on the board itself and drives a 7-inch 1024×600 MIPI-DSI panel, with the ESP32-P4’s Pixel Processing Accelerator scaling the original 320×200 output in hardware to hold 30 FPS.
Why does the AI get only a 51-ray depth scan instead of the game’s internal state?
Ricci deliberately limited what the model can see so it works from something close to a human player’s view. The first attempt used just seven rays, and narrow doorways fell between them, so Claude kept walking into walls. Raising the count to 51 made passages detectable without handing over the engine’s full state.
What will I learn if I build this?
You pick up hardware-accelerated display scaling on a modern MCU, lockstep protocol design over WebSockets, and how to build a perception abstraction that exposes only chosen data. The debugging lesson matters most: mismatched angle sign conventions between firmware and documentation are the same class of bug that breaks robotics telemetry and capstone sensor rigs.
