Google DeepMind has developed Gemini Robotics, a new family of AI models designed to bring advanced AI capabilities into the physical world through robotic systems. Building upon the foundation of Gemini 2.0, these models demonstrate significant advancements in embodied AI – the ability for AI systems to perceive, reason about, and interact with the physical environment. This report summarizes the key findings from Google DeepMind’s research, highlighting how these models are expanding the frontier of robotic capabilities through improved embodied reasoning, dexterous manipulation, and adaptability.

Introduction

Recent advances in large multimodal models have demonstrated remarkable capabilities in digital domains, but translating these capabilities to physical agents remains challenging. Robots need to perceive their surroundings, reason about physical relationships, and interact with objects safely and competently. The Gemini Robotics family represents a significant step toward bridging this gap by extending Gemini 2.0’s multimodal reasoning capabilities into the physical world.

This development suggests a potential shift in robotics: from specialized systems built for specific tasks to general-purpose robots with foundational understanding of physical interactions. With such advances, robots could potentially understand and execute a broader range of tasks through natural language instructions without requiring specialized programming.

Embodied Reasoning Capabilities

Embodied reasoning refers to an AI system’s ability to understand and reason about physical space, object relationships, and potential interactions. The researchers introduced a new benchmark called ERQA (Embodied Reasoning Question Answering) to evaluate these capabilities across multiple categories including spatial reasoning, trajectory reasoning, and multi-view reasoning.

Gemini 2.0 already demonstrated strong embodied reasoning capabilities, which were further enhanced in Gemini Robotics-ER. These capabilities include:

  • 2D Object Detection: Open-vocabulary detection of objects with precise bounding boxes
  • 2D Pointing: Identifying specific points on objects, object parts, or free space based on natural language descriptions
  • Trajectory Prediction: Generating motion paths between points
  • Grasp Prediction: Determining optimal grasp points and orientations for robotic manipulation
  • Multi-view Correspondence: Understanding 3D scenes by correlating points across different camera views
  • 3D Detection: Predicting metric 3D bounding boxes from single images

These capabilities enable zero-shot and few-shot robot control through code generation and in-context learning, allowing the models to perform robot control tasks without explicit training on robot action data.

Gemini Robotics: A Vision-Language-Action Model

Building on the foundation of Gemini Robotics-ER, the researchers developed Gemini Robotics, a Vision-Language-Action (VLA) model that directly generates robot actions in response to visual inputs and language instructions. Key features include:

  • Architecture: A cloud-based backbone combined with a local action decoder running on the robot’s computer for low-latency control
  • Training Data: Thousands of hours of expert demonstrations across diverse tasks, combined with non-action data like web documents, code, and visual content
  • Performance: The model can complete a variety of dexterous manipulation tasks out-of-the-box, significantly outperforming baseline models, particularly for deformable object manipulation

The model demonstrates strong generalization capabilities across three critical dimensions:

  1. Visual Generalization: Maintaining performance despite changes in background, lighting, or the presence of distractor objects
  2. Instruction Generalization: Understanding variations in language instructions, including different phrasings, languages, and levels of specificity
  3. Action Generalization: Adapting learned movements to new object positions or different object instances

Specialization and Adaptation

The researchers explored how Gemini Robotics can be specialized for particularly challenging tasks or adapted to new scenarios:

Long-horizon Dexterity

Through fine-tuning with high-quality demonstration data, Gemini Robotics can master extremely complex, long-horizon tasks requiring precise dexterity, including:

  • Making an origami fox with multiple precise folds
  • Packing a lunch-box with multiple items requiring different manipulation skills
  • Playing card games requiring fine manipulation of thin objects
  • Using tools like tongs and spoons to manipulate food items

The specialized models achieved an average success rate of 79% across these challenging tasks, with 100% success on the complete lunch-box packing sequence.

Enhanced Reasoning

By bridging action prediction more closely with embodied reasoning capabilities, the researchers created a variant with improved performance on tasks requiring more abstract reasoning and generalization. This version excelled at tasks requiring one-step reasoning, semantic understanding, and spatial comprehension.

Fast Adaptation

The model demonstrated the ability to learn new short-horizon tasks with remarkably little demonstration data – achieving over 70% success rates on 7 out of 8 evaluated tasks with at most 100 demonstrations. For some tasks, it reached 100% success rate with this limited training.

Adaptation to New Embodiments

Preliminary experiments showed that Gemini Robotics could be efficiently adapted to control entirely different robot platforms, including a bi-arm Franka robot with parallel grippers and the Apollo humanoid robot with five-fingered hands. The adapted models maintained the core generalization capabilities of the original system.

Responsible Development and Safety

The researchers emphasized their commitment to responsible AI development, acknowledging both traditional physical safety concerns in robotics and new considerations specific to AI-driven systems:

  • Physical Action Safety: Ensuring robots respect hard physical constraints like obstacle avoidance and force regulation
  • Content Safety: Inheriting Gemini safety policies to prevent harmful conversational content
  • Semantic Action Safety: Understanding implicit safety constraints in unstructured environments (e.g., not placing soft toys on hot stoves)

The models were evaluated on safety benchmarks including ASIMOV-datasets, with results showing strong semantic understanding of physical safety scenarios.

Limitations and Future Work

Despite significant advances, several challenges remain:

  • Improving numerical prediction precision for fine-grained control
  • Enhancing abilities to handle complex scenarios requiring both multi-step reasoning and precise movements
  • Reducing the data needed to adapt to new robot embodiments
  • Working toward zero-shot cross-embodiment transfer

Conclusion

The Gemini Robotics family represents a substantial step toward general-purpose autonomous AI in the physical world. By leveraging the world knowledge and reasoning capabilities of large multimodal models and grounding them in physical actions, these systems demonstrate a path toward more adaptable and capable robotic systems.

The combination of strong embodied reasoning, dexterous manipulation, and adaptability suggests a future where robots could potentially understand and execute a much broader range of tasks through natural language instructions without requiring specialized programming. As this technology matures, it holds promise for expanding robotic applications across domains, while raising important considerations about safety and societal implications that will need to be carefully addressed.