The Gemma 4 Good Hackathon

The Gemma 4 Good Hackathon · Main Track · First prize

GEM-4: Gemma Embodied 4 Physical Assistance

Built by Takaki Maeda, Kazuma Hayakawa, Gayagayagaya4, Yuta_soyokaze, and Yushun

A wearable robot-arm research prototype using Gemma 4 E2B, trained on robot datasets and teleoperated episodes. Its first-person video pipeline estimates wrist motion; conversion into robot-training episodes remains incomplete.

I also participated in this hackathon with Memory Moment. This coverage was prepared independently after results were announced.

The idea worth stealing

Build toward training robots from human video.

Teleoperation collects one episode at a time, and daily assistance is an open-ended list of small actions. GEM-4 films a person doing the task with a head-mounted GoPro, estimates the wrist from hand landmarks and metric depth, and is meant to convert the clip into robot training data; a QLoRA-tuned Gemma 4 already writes the subtask labels for robot episodes. The action model itself cross-attends to 18 evenly spaced layers of Gemma 4 E2B in the shipped configs, so the arm borrows the model’s understanding of a scene instead of learning it from scratch.

What we checked

Date checked: .

Public repository at the submission revision, Kaggle writeup, and linked artifacts; code was read, not executed. Deployment checks and limits are described below.

Labels apply to each finding and the evidence described, not to the project as a whole.

Not independently verified means the available evidence was insufficient to confirm a claim. It does not mean the claim is false.

  1. The action head reads 18 of Gemma’s 35 layers

    Verified

    Every shipped config sets 18 action-head blocks in “even” mode, so each block cross-attends to one evenly spaced Gemma 4 E2B layer, and a rank-64 LoRA tunes Gemma itself. “Each layer” is the design; this is the config. Source

  2. The checkpoints are public, the LIBERO numbers are theirs

    Partially verified

    Six fine-tuned checkpoints and the OXE pretrain base went to Hugging Face on May 18, with matching eval configs. We could not run the simulator, so 72, 92, 89, and 43 percent stay the builders’ numbers. Source

    What we need: Reproducible LIBERO runs with checkpoint hashes, evaluation configs, seeds, and per-task outputs.

  3. The robot has not yet learned from people

    Partially verified

    The hand pipeline turns GoPro video into wrist pose and pinch distance, but nothing converts that into training episodes. The three robot datasets were teleoperated through MimicRec. Source

    What we need: The video-to-training-episode converter, one exported episode, and a training run using it.

Under the hood

Boundaries without a model
MimicAnno cuts episodes at gripper open–close events, end-effector velocity valleys, and action-norm change points, tracks each segment with SAM 3, then asks a QLoRA-tuned Gemma 4 E4B or 26B for the label.
Built on VLA-Adapter and X-VLA
The Prismatic code is a slimmed vendoring of VLA-Adapter, with X-VLA’s per-domain projectors; each real-robot fine-tune adds a fourteenth domain row initialized at random, a rule written down after getting it wrong.

Nuance

The published training setup uses teleoperated episodes plus OXE and LIBERO; training on human video is the next step, not the shipped one. Cross-embodiment transfer is not independently verified. The builders report 43 percent on LIBERO-Long and a fit for one user. “Hey GEM” is heard by Whisper on a Jetson. We ran none of the tests.

Sources

Last updated: . Editorial updates do not imply a new technical check.

Builder credits reviewed: , using the official announcement, submission, and any linked credit sources.

Takaki Maeda is credited as takaki-maeda on Kaggle and TakakiMaeda on the linked GitHub profile. Gayagayagaya4, Yuta_soyokaze, and Yushun are the available public credits; full names were not confirmed.

Builder? Add context, request a correction, or ask for a re-check →