A Raspberry Pi 5 can now hold a spoken conversation with nothing on the other end of the Wi-Fi. Google’s engineers and the Raspberry Pi team spent months tuning LiteRT, Google’s on-device inference runtime, so Gemma language models run on the Pi’s own CPU and GPU. No API key, no monthly bill, no round trip to a data centre.
What the demo actually does
The showcase build is Reachy Mini, a small desk robot whose only brain is a Pi 5. It hears you, reasons about what you said with Gemma, and answers with speech and movement, entirely offline. For a capstone or thesis team that flips a familiar problem: your robot keeps working when campus Wi-Fi dies during the demo, and none of your lab audio ever leaves the bench.
The numbers behind it
On an 8GB Pi 5 with four CPU threads, LiteRT-LM runs Gemma 4 E2B at 99 tokens/sec prefill and 9 tokens/sec decode, peaking at 1432 MB of RAM. The same model under llama.cpp (Q4_0) manages 24 and 4 tokens/sec while eating 4406 MB. That is roughly 4x the prefill speed in a third of the memory, which is the difference between a robot that answers and one that swaps to disk mid-sentence. Gemma’s tokenizer averages about 4.2 characters per token, so the voice demo streams around 27.3 characters per second, near 300 words per minute, or twice normal speaking pace.
Smaller options exist if 1432 MB is too much for your build. Gemma 3 270M suits sentiment analysis or entity extraction, EmbeddingGemma 300M handles on-device semantic search and RAG, and Gemma 3 1B covers multilingual text without the vision layers. The Pi 5‘s quad-core Cortex-A76 supplies roughly 153.6 GFLOPS before the VideoCore GPU path is even brought in.
Build it yourself
- Raspberry Pi 5, 8GB variant, plus the active cooler. Sustained inference will thermal-throttle a bare board.
- NVMe HAT or a fast microSD. Model files run to a few gigabytes and load time is dominated by storage speed.
- A USB or I2S microphone. I2S mics wire to the Pi’s GPIO header on pins 12, 35 and 38, and need
dtoverlaylines inconfig.txtbefore ALSA will see them. - The
litert-lmruntime and a.litertlmmodel file, which you pull from the LiteRT community page on Hugging Face.
Start with Gemma 3 270M to confirm your audio chain works before downloading a multi-gigabyte E2B file. Full benchmark tables and setup steps are on the Raspberry Pi announcement: Mastering edge AI on Raspberry Pi with LiteRT and Gemma.
Frequently Asked Questions
How fast does Gemma actually run on a Raspberry Pi 5?
On an 8GB Pi 5 using four CPU threads, LiteRT-LM runs Gemma 4 E2B at 99 tokens/sec prefill and 9 tokens/sec decode, peaking at 1432 MB of RAM. That works out to roughly 300 words per minute of generated speech, about twice normal talking speed.
What hardware do I need beyond the Pi itself?
An 8GB Raspberry Pi 5 with the active cooler, fast storage (NVMe HAT or a quick microSD, since model files run to several gigabytes), and a USB or I2S microphone if you want voice input. I2S mics attach to the GPIO header and need dtoverlay lines in config.txt before ALSA detects them.
What will I learn if I build this?
You will pick up on-device model deployment, quantisation trade-offs and how to read a tokens/sec benchmark, plus practical Linux audio work with ALSA and GPIO-attached I2S microphones. Those are the same skills behind offline voice assistants, smart cameras and autonomous robot projects, and they make for a strong capstone or competition entry.
