Deployment Is Not Destiny: Robot Recomposition in the Field

Core concept: a robot is modified in the field with plug-and-play unseen payloads — **Core Concept:** Our framework allows robots to be quickly modified with plug-and-play *unseen payloads* for ad hoc tasks. Non-roboticist end-users can quickly reconfigure a robot in the field without reprogramming, and robots require no prior knowledge of introduced payloads. Within minutes, a robot can compose new capabilities into goal-directed task plans.

Abstract

The tight coupling of subsystems in most robots, though a natural consequence of their complexity, leads to monolithic designs that are time-consuming and difficult to adapt after initial deployment. To address this challenge, we present a framework and supporting abstractions for recomposition during runtime that enable robots to quickly integrate previously unseen modular software, hardware, and compute payloads. Our approach allows non-expert users to quickly add new capabilities in the field through a true plug-and-play process. Crucially, new resources are not only immediately available to a host robot but are also shared with distributed peers, enabling compute-constrained systems to access powerful new remote capabilities. Our framework reduces reconfiguration time to a matter of minutes with no developer intervention, in stark contrast to the hours of expert effort often required for traditional manual integration. We demonstrate our method in two disaster response scenarios, including radioactive source localization at an operational nuclear reactor facility and a thermal-guided search for people in dark, difficult-to-reach spaces. These demonstrations show how in-field recomposition provides timely, flexible, and accessible adaptation to dynamic requirements, representing a critical step toward creating robots that can quickly evolve alongside the tasks, technologies, and environments they support.

Approach

Our framework is built around the principle of compositionality: every previously unseen payload is treated as a self-contained component that can be dynamically integrated into the robot's decision-making and execution pipeline. Existing low-level interfaces for each component type — software, hardware, and compute — are unified and extended where required to produce similar composable properties. A dedicated Component Manager combines these interfaces into a top-level abstraction that handles component discovery and utilization during runtime, with no developer intervention and minimal system downtime.

Framework overview: initial components, component manager, system-level and task-level behavior trees — **Framework Overview:** The system begins with a set of software, hardware, and compute *initial components*. Each software component affords a set of behaviors that are composed into a *system-level behavior tree* responsible for processing system updates and task planning. A dedicated *component manager* is responsible for discovery and management of unseen *added components* introduced during runtime. When provided a task goal, the robot leverages available components — including those introduced during runtime — to generate and execute a *task-level behavior tree*. Behaviors afforded by each component are indicated with color and wires showing origin.

Component Abstractions

Each component type requires a different abstraction, but all three are unified so that they expose the same composable properties to the planner and to distributed peers.

Software Containers + databases

Software payloads ship a containerized image plus a keyword-indexed relational database holding compiled behavior and interface libraries for multiple CPU architectures and PDDL action, domain, and problem fragments describing the behaviors the payload affords. Payloads arrive on USB flash drives and are discovered via udev events.

Hardware USB + rail mounts

Hardware payloads connect over USB and are physically mounted with a rail-based mounting plate for fast recomposition. Non-block USB devices are automatically mapped into the containers of concurrently introduced driver software, giving plug-and-play access to sensors and actuators.

Compute P2P discovery + RPC

Compute payloads join over Ethernet or WiFi on a predetermined subnet, discover peers via UDP multicast, and exchange data over TCP. Peers trade component databases on discovery, so a compute-constrained robot can plan and execute using capabilities hosted elsewhere.

Component Manager

Every robot and external compute payload runs a Component Manager that owns the lifetime of all other components and reacts to three classes of runtime update: USB block device updates (scan for valid software payloads, verify CPU architecture, disk, and GPU memory, then load images and copy databases), new device nodes (map hardware into the matching driver container), and peer signaling (exchange databases with newly discovered peers and prune on disconnect). The system-level behavior tree monitors these updates and re-queries all stored databases, so plans always reflect the current capability set — local and remote.

Demonstrations

We present demonstrations spanning three host robots and a diverse set of unseen payloads. Together they validate four key attributes of plug-and-play recomposition during runtime.

A1

Incremental Payload Integration

Adding and removing payloads to provide new capabilities as mission needs evolve.

A2

Distributed Capability Sharing

Sharing newly introduced local payloads and utilizing remote payloads hosted on another peer system.

A3

Host-Agnostic Integration

Reusing the same payloads across different host robots, tasks, and environments.

A4

Concert of Independently Designed Payloads

Cooperation between independently designed unseen payloads to achieve a provided task goal.

Host robots and unseen software, hardware, and compute payloads used in the demonstrations — **Host Robots and Unseen Payloads:** Each host robot and the external compute payload starts with a subset of the initial software components shown in the Framework Overview. Software is injected into host robots during operation via USB drives, each containing one or several software components. Hardware payloads are mounted during operation via a rail-based mounting system. Software payloads required to use hardware payloads are indicated. Host robots and payloads have unique colored identifying tags used throughout both demonstrations below.

Each host robot has a distinct computational profile: a Boston Dynamics Spot with an NVIDIA Jetson AGX Orin (64 GB shared RAM/GPU memory, 1 TB disk) in Demonstration 1; a Husarion Panther with an Intel i9-13900, 64 GB RAM, 1 TB disk, and no GPU; and a Robotis Turtlebot with a Raspberry Pi 4 Model B, 2 GB RAM, 32 GB disk, and no GPU. The external compute payload in Demonstration 2 is an identical Jetson AGX Orin.

Demonstration 1 — Radioactive Source Localization

Scenario. We mimic a partial blackout at an operational nuclear reactor during which an internal radiation alarm has been triggered. Demonstrators placed actual radioactive material out of plain sight, but in a location from which beta and gamma emissions would be detectable by instruments. A Spot host robot is deployed alongside a human teammate with a suite of unseen payloads for person tracking and radioactive source localization.

Watch on YouTube ↗

Stage I — RGB-D person following The teammate plugs in an RGB-D camera with supporting driver software, a YOLO image detection model, and a vision-based tracking controller, then issues the goal "follow me through the facility." The system-level behavior tree generates a task-level behavior tree for person following, letting the robot traverse the building without a prior map.
Stage II — Thermal person following in the dark A blackout kills all lights and the RGB-D camera becomes ineffective. The teammate swaps the camera and drivers for a thermal camera plus drivers and a monocular depth-estimation model, then repeats the same goal. The new thermal sensing modality composes with the already-running detection model and tracking controller to continue through the building.
Stage III — Radiation source localization On arriving at the contaminated room, the teammate removes the running payloads and plugs in a Compton radiation camera with drivers and a pan–tilt unit with laser plus drivers. The goal "point at the radiation source" yields a task-level behavior tree that detects radiation with the Compton camera and points at it with the PTU laser, successfully localizing the hidden source.

Demonstration 1: radioactive source localization, three stages with payloads, behavior trees, and sensor data — **Demonstration 1: Radioactive Source Localization** consisting of stage I) person-following traversal through the building using an RGB-D camera, stage II) continued traversal through a dark part of the facility using a thermal camera, and stage III) radiation source localization and indication with a Compton radiation camera and pan-tilt unit with laser. The top row shows the unseen payloads plugged into the host robot during each stage. The middle row shows the user-provided natural language goals during each stage and the corresponding autonomously generated task-level behavior trees. The bottom row shows images from third-person perspective and onboard sensor data collected and processed during task execution. Software and hardware tags match Host Robots and Unseen Payloads, and behaviors and robot data snapshots indicate the components that enable them. Recomposition time for each stage is shown above the top row.

Demonstration 2 — Thermal-Guided Person Search

Scenario. In a mock disaster-search task, a large Panther host robot is configured to search for people outdoors using a thermal camera, but is too large to fit through a discovered confined building entryway. A smaller Turtlebot host robot is powered on and, using resources shared from the external compute payload, enters the building and continues the search.