Embodied Task Decomposition

This skill decomposes high-level natural language instructions into atomic subtasks that a robot can execute.

When to Use

• User provides an image AND a task instruction
• User asks to "decompose", "break down", or "split" a task
• User wants step-by-step actions for robot execution

Input Format

1. Image: Photo of the physical scene (any environment: kitchen, office, outdoor, etc.)
2. Task Instruction: Natural language description of what to accomplish

Example:

Task Instruction: take toasted bread from bread machine on white table place on plate
Image: [image path or description]

Output Format

Numbered list of subtasks, each following format:

{action} {target} {location/optional prepositional phrase} with {left/right/either} gripper

Process

1. Analyze the image - Identify objects, surfaces, locations, tools visible
2. Understand the task - What is the goal? What needs to be moved/ manipulated?
3. Break into atomic actions - Each subtask = one action from the action bank
4. Specify gripper - Always indicate left, right or either gripper

Action Bank

Refer to action-bank.md for the complete list of allowed actions. All subtasks MUST use actions from this bank.

Examples

See examples.md for detailed decomposition examples across different domains.

Important Notes

• Use ONLY actions from the action bank
• Each subtask = one primary action
• Always specify gripper (left/right/either)
• Include target object and location
• Keep subtasks atomic and sequential
• Consider object state changes (e.g., "open bag" before "take fruit")

Updating the Action Bank

The agent MAY add new actions to the action bank when needed. To add a new action:

1. Check for duplicates - Search existing actions for similar functionality
2. Verify functional difference - New action must serve a distinct purpose
3. Add with documentation - Include description and example usage

Duplicate Check Criteria

A new action is considered a duplicate if it:

• Has the same name as an existing action
• Describes the same physical movement (e.g., "lift" vs "raise")
• Can be used interchangeably with an existing action in all contexts

Adding a New Action

When adding to action-bank.md, follow this format:

| action_name | Description | Example Usage |
|-------------|-------------|---------------|
| new_action | What it does | "new_action the object"

Example of adding "insert" (different from "place" - "place" = put on surface, "insert" = put into container):

| insert | Put object inside a container or slot | "insert the key into the lock"