LeRobot v0.4.0: Supercharging OSS Robot Learning

LeRobot v0.4.0 delivers a major upgrade for open-source robotics, introducing scalable Datasets v3.0, powerful new VLA models like PI0.5 and GR00T N1.5, and a new plugin system for easier hardware integration. The release also adds support for LIBERO and Meta-World simulations, simplified multi-GPU training, and a new Hugging Face Robot Learning Course.

• LeRobot v0.4.0: Supercharging OSS Robot Learning
• TL;DR
• Table-of-Contents
• Datasets: Ready for the Next Wave of Large-Scale Robot Learning
• Simulation Environments: Expanding Your Training Grounds
• Codebase: Powerful Tools For Everyone
• Policies: Unleashing Open-World Generalization
• Robots: A New Era of Hardware Integration with the Plugin System
• The Hugging Face Robot Learning Course
• Final thoughts from the team

We've completely overhauled our dataset infrastructure with LeRobotDataset v3.0, featuring a new chunked episode format and streaming capabilities. This is a game-changer for handling massive datasets like OXE (Open X Embodiment) and Droid, bringing unparalleled efficiency and scalability.

• Chunked Episodes for Massive Scale: Our new format supports datasets at the OXE-level (> 400GB), enabling unprecedented scalability.
• Efficient Video Storage + Streaming: Enjoy faster loading times and seamless streaming of video data.
• Unified Parquet Metadata: Say goodbye to scattered JSONs! All episode metadata is now stored in unified, structured Parquet files for easier management and access.
• Faster Loading & Better Performance: Experience significantly reduced dataset initialization times and more efficient memory usage.

We've also provided a conversion script to easily migrate your existing v2.1 datasets to the new v3.0 format, ensuring a smooth transition. Read more about it in our previous blog post. Open-source robotics keeps leveling up!

Working with LeRobot datasets just got a whole lot easier! We've introduced a powerful set of utilities for flexible dataset editing.

With our new lerobot-edit-dataset CLI, you can now:

• Delete specific episodes from existing datasets.
• Split datasets by fractions or episode indices.
• Add or remove features with ease.
• Merge multiple datasets into one unified set.

# Merge multiple datasets into a single dataset.
lerobot-edit-dataset \
--repo_id lerobot/pusht_merged \
--operation.type merge \
--operation.repo_ids "['lerobot/pusht_train', 'lerobot/pusht_val']"
# Delete episodes and save to a new dataset (preserves original dataset)
lerobot-edit-dataset \
--repo_id lerobot/pusht \
--new_repo_id lerobot/pusht_after_deletion \
--operation.type delete_episodes \
--operation.episode_indices "[0, 2, 5]"

These tools streamline your workflow, allowing you to curate and optimize your robot datasets like never before. Check out the docs for more details!

We're continuously expanding LeRobot's simulation capabilities to provide richer and more diverse training environments for your robotic policies.

LeRobot now officially supports LIBERO, one of the largest open benchmarks for Vision-Language-Action (VLA) policies, boasting over 130 tasks! This is a huge step toward building the go-to evaluation hub for VLAs, enabling easy integration and a unified setup for evaluating any VLA policy.

We've integrated Meta-World, a premier benchmark for testing multi-task and generalization abilities in robotic manipulation, featuring over 50 diverse manipulation tasks. This integration, along with our standardized use of gymnasium ≥ 1.0.0 and mujoco ≥ 3.0.0, ensures deterministic seeding and a robust simulation foundation.

We're making robot control more flexible and accessible, enabling new possibilities for data collection and model training. Getting data from a robot to a model (and back!) is tricky. Raw sensor data, joint positions, and language instructions don't match what AI models expect. Models need normalized, batched tensors on the right device, while your robot hardware needs specific action commands.

We're excited to introduce Processors: a new, modular pipeline that acts as a universal translator for your data. Think of it as an assembly line where each ProcessorStep handles one specific job—like normalizing, tokenizing text, or moving data to the GPU.

Training large robot policies just got a lot faster! We've integrated Accelerate directly into our training pipeline, making it incredibly simple to scale your experiments across multiple GPUs with just one command:

accelerate launch \
--multi_gpu \
--num_processes=$NUM_GPUs \
$(which lerobot-train) \
--dataset.repo_id=${HF_USER}/my_dataset \
--policy.repo_id=${HF_USER}/my_trained_policy \
--policy.type=$POLICY_TYPE

In a major milestone for open-source robotics, we've integrated pi0 and pi0.5 policies by Physical Intelligence into LeRobot! These Vision-Language-Action (VLA) models represent a significant leap towards addressing open-world generalization in robotics.

In another exciting development, we've integrated NVIDIA's GR00T N1.5 into LeRobot, thanks to a fantastic collaboration with the NVIDIA robotics team! This open foundation model is a powerhouse for generalized robot reasoning and skills. As a cross-embodiment model, it takes multimodal input to perform complex manipulation tasks in diverse environments.