Smol2Operator: Post-Training GUI Agents for Computer Use

**TL;DR:**This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection here.

• Introduction
• 1. Data Transformation and Unified Action Space
• 2. Phase 1: From Zero to Perception
• 3. Phase 2: From Perception to Cognition
• 4. All you need is Open Source
• 5. Conclusion
• What's Next?

Graphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.

In this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.

Rather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.

GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, …

Our approach leverages SmolVLM2-2.2B-Instruct as the baseline model, a small powerful vision-language model that initially has no grounding capabilities for GUI tasks. This makes it an ideal candidate to demonstrate the effectiveness of our training methodology. Through our two-phase training process, we first instill grounding capabilities in the model, then enhance it with agentic reasoning abilities using Supervised Fine-Tuning (SFT).

We evaluate our approach on an established perception benchmark: ScreenSpot-v2, which tests the model’s ability to understand and locate elements within screenshots. Our process is inspired by the AGUVIS paper, and we leverage their carefully curated datasets to build upon their foundational work.

Evolution of ScreenSpot-v2 performance during the training phase of the base model SmolVLM2-2.2B-Instruct.

This section explains how we convert heterogeneous GUI actions format from multiple datasets into a single unified format. By standardizing function names, signatures, and parameters, we create consistent, high-quality data that forms the foundation for effective model training.

One of the primary challenges when working with multiple GUI automation datasets is the lack of standardization in action representations. Different datasets use varying function signatures, parameter naming conventions, and action taxonomies, making it difficult to train a unified model across diverse data sources.

We took the open-source datasets (xlangai/aguvis-stage1, xlangai/aguvis-stage2), originally used by AGUVIS, and implemented a comprehensive data transformation pipeline to create a unified action space. Our approach involved:

Function Parsing and Normalization: We developed a function parser (seeutils/function_parser.py) that can extract and parse function calls from various formats across all datasets. This parser supports any function signature format, handles complex parameter structures, and can reconstruct function calls with proper parameter ordering.Action Space Unification: We implemented a comprehensive action conversion system (seepreprocessing/action_conversion.py) that transforms all original action representations into a standardized function naming and argument structure. This process highlighted the significant inconsistencies in function signatures across different datasets and allowed us to:

• Remove undesired or redundant actions
• Standardize parameter naming conventions
• Create a cohesive action vocabulary

(Bonus) Flexible Adaptation Framework: Our transformation pipeline includes utilities that allow users to:

• Adapt the entire dataset to their own action space naming conventions using the utils/action_space_converter.py tool
• Extract and analyze the current action space structure

Here are real examples from our action conversion system (preprocessing/action_conversion.py) showing how we transform heterogeneous action representations into our unified format (grounding coordinates normalized to [0,1]):

Before (Original Action Dataset Formats):

# Mobile Actions
mobile.home()
mobile.open_app(app_name='drupe')
mobile.swipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])
mobile.long_press(x=0.799, y=0.911)
mobile.terminate(status='success')
# Desktop Actions
pyautogui.click(x=0.8102, y=0.9463)
pyautogui.doubleClick(x=0.8102, y=0.9463)
pyautogui.hotkey(keys=['ctrl', 'c'])
pyautogui.scroll(page=-0.1)
pyautogui.write(message='bread buns')
pyautogui.dragTo(from_coord=[0.87, 0.423], to_coord=[0.8102, 0.9463])

After (Unified Action Dataset Formats):

# Unified Mobile Actions
navigate_home()
open_app(app_name='drupe')
swipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])
long_press(x=0.799, y=0.911)
final_answer('success')
# Unified Desktop Actions
click(x=0.8102, y=0.9463)
double_click(x=0.8102, y=0.9463)
press(keys=['ctrl', 'c'])
scroll(direction='up', amount=10) # Smart direction detection
type(text='bread buns')
drag(from_coord=[0.87, 0.423], to_coord=[0.8102, 0.9463])

This unification process was essential for creating coherent training data that allows the model to learn consistent action patterns across diverse GUI environments.

To maximize flexibility for different use cases, we developed the Action Space Converter (utils/action_space_converter.py), a tool that allows users to easily adapt from an action space to their own custom action vocabularies and naming conventions.

You can use this tool to transform one action signature (function names, parameter names, and parameter value changes, ...) into another:

Before

assistant_message: "Action: click(x=0.5, y=0.3)"

After

assistant_message: "Action: touch(x_coord=200, y_coord=300)"