Building Herbie — A VLM-Powered Social Robot

Building a robot that can talk, see, move, and interact naturally with humans is one of the most exciting challenges in physical AI. Herbie is my attempt at exactly that — a social robot powered by vision-language models (VLMs) designed for educational environments.

Here is the final Herbie robot in action — walking, talking, and responding to voice commands:

This work presents a modular physical AI robot architecture that combines natural-language interaction, visual perception, high-level task planning, task delegation, autonomous navigation, and robotic manipulation. The architecture separates cognitive reasoning from physical control, allowing the system to interpret user requests while executing real-world tasks through specialized agents and hardware modules.

A. System Overview

Herbie accepts a user task through natural-language instruction and interprets it using the Main Agent. The Main Agent performs intent understanding, high-level planning, reasoning, and task decomposition. It determines whether a request requires navigation, manipulation, or a combination of both activities.

After task decomposition, the system delegates the required task to specialized sub-agents. The navigation agent manages movement between locations, while the manipulation agent handles object detection, grasping, and object-handling operations. The physical execution layer converts the delegated tasks into low-level actions using a Vision-Language-Action model, motion generation, and low-level control modules.

B. Cognitive Planning and Physical Execution

The cognitive layer is represented by the Main Agent, which uses visual and language information to understand the robot's environment and user requests. The high-level planner produces an action plan by identifying the task goal, relevant objects, target locations, and task constraints. The reasoning and task-decomposition component divides complex instructions into smaller executable actions.

The physical execution layer receives the planned task and translates it into robot movements. The VLA module combines visual context with the planned instruction to generate action-oriented outputs. The motion generator produces appropriate motion sequences, while the low-level control module sends commands to the robotic base, arms, and gripper. Vision input is available to both the Main Agent and physical execution layer, enabling context-aware planning and closed-loop action execution.

C. Multi-Agent Task Delegation

The task delegator acts as an intermediate coordination component between the Main Agent and the sub-agents. It receives a structured task generated by the Main Agent and selects the appropriate execution agent. A task object contains the task goal, target object, target location, execution priority, and operational constraints.

For navigation tasks, the task delegator sends the required destination and movement constraints to the navigation agent. The navigation agent uses environmental perception and obstacle information to plan safe robot movement. For manipulation tasks, the manipulation agent receives information about the target object and desired action, such as grasping, picking, placing, or delivering an object.

The sub-agents return task progress, completion state, and failure information to the task delegator and Main Agent. This feedback enables the robot to inform the user about the current status and allows the Main Agent to modify or repeat a task if execution fails.

D. Hardware and Control Integration

The hardware layer connects the intelligent software modules with the robot's physical components. The onboard computer operates as the central processing unit and communicates with the vision system, obstacle-detection unit, communication interfaces, robotic arms, motor-control unit, and speech unit. The power-supply unit provides regulated power to the onboard computer and connected hardware modules.

The vision system provides camera-based environmental information for object recognition, scene understanding, and spatial reasoning. Based on the generated control commands, the motor-control unit drives the wheel base for navigation. The robotic arms perform manipulation actions, while the speech unit provides voice-based interaction and task-status updates to the user.

Simulation Environment

Before testing on the physical robot, I built a complete 3D simulation that mirrors the robot's API exactly. The simulation runs in the browser using WebGL and WebSockets, and exposes the same endpoints as the physical robot server. This made it possible to develop and test the AI agent, navigation logic, and control flows entirely in software before deploying to hardware.

The simulation was invaluable for iterating on the vision agent loop — the system that lets Herbie see its surroundings, make decisions, and execute physical commands in a observe-think-act cycle.

Key Takeaways

Building Herbie taught me that the hardest part of robotics is not the hardware or the AI individually, but the integration between them. Getting an LLM to produce valid movement commands, getting speech synthesis to complete before the next response arrives, and getting the vision loop to converge without getting stuck — each of these required careful architecture and a lot of iteration.

The full source code is available on GitHub for anyone interested in diving deeper.