UI Agent Patterns

Patterns for orchestrating AI agents to generate, refine, and maintain user interfaces. This skill bridges Karpathy's "new programming vocabulary" with practical UI/UX development workflows.

When to Use This Skill

• Delegating complex UI generation to specialized agents

• Deciding between synthesis-master vs specialized agent architectures

• Orchestrating multi-agent workflows for design systems

• Managing handoffs between research, design, and implementation agents

• Building agent pipelines for iterative UI refinement

• Scaling UI generation beyond single-agent capabilities

Core Concepts

The New Programming Vocabulary

Karpathy's insight: LLMs introduce new programming primitives that extend beyond functions and objects:

Primitive Description UI Application

Agents Autonomous LLM-powered workers UI generators, reviewers, refiners

Subagents Delegated specialists Component builders, accessibility checkers

Prompts Instructions as code Design specifications, component contracts

Contexts Shared state and knowledge Design tokens, brand guidelines

Memory Persistent learning Style preferences, past decisions

Modes Behavioral configurations Draft mode, production mode, audit mode

Permissions Capability boundaries Read-only review vs code modification

Tools External capabilities Figma API, browser DevTools, screenshot capture

Plugins Modular extensions Design system loaders, component libraries

Skills Reusable knowledge This file - codified expertise

Hooks Lifecycle interceptors Pre-commit design checks, post-render audits

MCP Model Context Protocol Tool integration standard

Workflows Orchestrated sequences Design-to-code pipelines

Agent Architecture Patterns

Pattern 1: Synthesis-Master Architecture

A single powerful agent handles the full UI generation task.

When to Use:

• Simple, well-defined UI tasks

• Tight coupling between decisions

• Speed is critical

• Context window sufficient for entire task

Structure:

[User Request] | v +------------------+ | Synthesis-Master | | (Full Context) | +------------------+ | v [Complete UI Output]

Implementation:

class SynthesisMasterAgent: """ Single agent handling all UI generation aspects. Best for: Landing pages, simple forms, atomic components """

def __init__(self, model: str = "claude-sonnet-4-5-20250929"):
    self.context = {
        "design_tokens": load_design_tokens(),
        "brand_guidelines": load_brand_context(),
        "component_library": load_component_docs(),
        "accessibility_rules": load_a11y_rules(),
    }

async def generate(self, request: UIRequest) -> UIOutput:
    prompt = f"""
    You are a senior UI engineer and designer. Generate a complete,
    production-ready component based on this request.

    Context:
    - Design Tokens: {self.context['design_tokens']}
    - Brand Guidelines: {self.context['brand_guidelines']}

    Request: {request.description}

    Output requirements:
    1. React/TypeScript component
    2. Tailwind CSS styling
    3. Accessibility attributes
    4. Responsive breakpoints
    5. Dark mode support
    """

    return await self.model.generate(prompt)

Advantages:

• Simpler orchestration

• No handoff overhead

• Consistent voice/style

• Lower latency

Disadvantages:

• Context window limits

• Single point of failure

• Hard to scale complexity

• No specialized expertise

Pattern 2: Specialized Agent Swarm

Multiple specialized agents collaborate on UI tasks.

When to Use:

• Complex design systems

• Tasks requiring different expertise

• Parallel processing beneficial

• Quality through specialization

Structure:

[User Request] | v +------------------+ | Orchestrator | +------------------+ | +-----------------+----------------+----------------+ | | | | v v v v +----------+ +----------+ +----------+ +----------+ | Research | | Design | | Code | | Review | | Agent | | Agent | | Agent | | Agent | +----------+ +----------+ +----------+ +----------+ | | | | v v v v [Context] [Wireframe] [Component] [Audit]

Specialized Agent Definitions:

Agent 1: Research Agent

class UIResearchAgent: """ Gathers context and prior art before design begins. """

permissions = ["read_codebase", "search_web", "read_figma"]

async def research(self, request: UIRequest) -> ResearchContext:
    return {
        "existing_patterns": await self.find_similar_components(),
        "competitive_analysis": await self.analyze_competitors(),
        "user_research": await self.gather_user_insights(),
        "technical_constraints": await self.identify_constraints(),
    }

Agent 2: Design Agent

class UIDesignAgent: """ Produces design specifications and wireframes. """

permissions = ["generate_images", "access_design_tokens"]

async def design(self, context: ResearchContext) -> DesignSpec:
    return {
        "layout": await self.generate_layout(),
        "spacing": await self.calculate_spacing(),
        "typography": await self.select_typography(),
        "colors": await self.derive_color_scheme(),
        "interactions": await self.define_interactions(),
    }

Agent 3: Implementation Agent

class UIImplementationAgent: """ Translates designs into production code. """

permissions = ["write_code", "access_component_library"]

async def implement(self, spec: DesignSpec) -> CodeOutput:
    return await self.generate_component(
        framework="react",
        styling="tailwind",
        typescript=True,
        spec=spec
    )

Agent 4: Review Agent

class UIReviewAgent: """ Audits output for quality, accessibility, and standards. """

permissions = ["read_code", "run_tests", "access_browser"]
mode = "audit"  # Read-only, cannot modify

async def review(self, code: CodeOutput) -> ReviewReport:
    return {
        "accessibility": await self.audit_a11y(),
        "performance": await self.audit_performance(),
        "design_fidelity": await self.compare_to_spec(),
        "code_quality": await self.lint_and_analyze(),
    }

Pattern 3: Hierarchical Delegation

Master agent delegates to subagents for specific subtasks.

When to Use:

• Complex pages with many components

• Need for parallel component generation

• Different components require different expertise

Structure:

[User Request: "Create a dashboard"] | v +------------------+ | Master Agent | | (Task Planning) | +------------------+ | +--------+--------+--------+ | | | | v v v v [Header] [Sidebar] [Charts] [Tables] Subagent Subagent Subagent Subagent | | | | v v v v [JSX] [JSX] [JSX] [JSX] | v +------------------+ | Master Agent | | (Integration) | +------------------+ | v [Complete Dashboard]

Implementation:

class HierarchicalUIOrchestrator: """ Master agent that delegates to specialized subagents. """

def __init__(self):
    self.subagents = {
        "header": HeaderComponentAgent(),
        "sidebar": SidebarComponentAgent(),
        "charts": DataVisualizationAgent(),
        "tables": DataTableAgent(),
        "forms": FormBuilderAgent(),
    }

async def generate_page(self, request: PageRequest) -> PageOutput:
    # Step 1: Plan the page structure
    plan = await self.plan_page_structure(request)

    # Step 2: Delegate component generation in parallel
    component_tasks = []
    for component in plan.components:
        agent = self.subagents[component.type]
        task = agent.generate(component.spec)
        component_tasks.append(task)

    components = await asyncio.gather(*component_tasks)

    # Step 3: Integrate components into cohesive page
    page = await self.integrate_components(components, plan.layout)

    # Step 4: Final coherence review
    return await self.ensure_coherence(page)

async def plan_page_structure(self, request: PageRequest) -> PagePlan:
    """
    Master agent determines page structure and delegation.
    """
    prompt = f"""
    Analyze this page request and create a component breakdown:

    Request: {request.description}

    For each component, specify:
    1. Component type (header, sidebar, chart, table, form, etc.)
    2. Component requirements
    3. Data dependencies
    4. Layout position

    Return as structured JSON.
    """
    return await self.model.generate(prompt, format="json")

Multi-Agent Workflow Patterns

Workflow 1: Design-to-Code Pipeline

Sequential workflow from design intent to production code.

class DesignToCodePipeline: """ Complete workflow from natural language to deployed UI. """

stages = [
    ("interpret", InterpretationAgent()),    # NL -> Design Intent
    ("design", DesignAgent()),               # Intent -> Wireframe
    ("specify", SpecificationAgent()),       # Wireframe -> Spec
    ("implement", ImplementationAgent()),    # Spec -> Code
    ("review", ReviewAgent()),               # Code -> Audit
    ("refine", RefinementAgent()),           # Audit -> Final Code
]

async def run(self, request: str) -> CodeOutput:
    context = {"request": request}

    for stage_name, agent in self.stages:
        result = await agent.process(context)
        context[stage_name] = result

        # Allow early exit on critical issues
        if result.has_blocking_issues:
            return self.handle_blocker(stage_name, result)

    return context["refine"]

Workflow 2: Iterative Refinement Loop

Agent loop that refines UI through multiple passes.

class IterativeRefinementWorkflow: """ Generate -> Review -> Refine loop until quality threshold met. """

def __init__(self, max_iterations: int = 5):
    self.generator = UIGeneratorAgent()
    self.reviewer = UIReviewerAgent()
    self.refiner = UIRefinerAgent()
    self.max_iterations = max_iterations
    self.quality_threshold = 0.85

async def run(self, request: UIRequest) -> RefinedOutput:
    # Initial generation
    current = await self.generator.generate(request)

    for iteration in range(self.max_iterations):
        # Review current version
        review = await self.reviewer.review(current)

        # Check if quality threshold met
        if review.score >= self.quality_threshold:
            return current

        # Refine based on feedback
        current = await self.refiner.refine(
            current=current,
            feedback=review.feedback,
            priority=review.critical_issues
        )

    # Return best effort after max iterations
    return current

Workflow 3: Parallel Variant Generation

Generate multiple design variants for comparison.

class ParallelVariantWorkflow: """ Generate multiple design variants in parallel for A/B consideration. """

async def generate_variants(
    self,
    request: UIRequest,
    variant_count: int = 3
) -> list[DesignVariant]:

    # Define variant strategies
    strategies = [
        {"style": "minimal", "focus": "whitespace"},
        {"style": "bold", "focus": "typography"},
        {"style": "playful", "focus": "interactions"},
    ][:variant_count]

    # Generate in parallel
    tasks = [
        self.generate_variant(request, strategy)
        for strategy in strategies
    ]

    variants = await asyncio.gather(*tasks)

    # Score and rank variants
    scored = await self.score_variants(variants, request.criteria)

    return sorted(scored, key=lambda v: v.score, reverse=True)

Agent Memory Patterns

Pattern: Design Decision Memory

Persist design decisions for consistency across sessions.

class DesignMemory: """ Persistent memory of design decisions and preferences. """

def __init__(self, project_id: str):
    self.project_id = project_id
    self.decisions = self.load_decisions()

def remember_decision(self, decision: DesignDecision):
    """
    Store a design decision for future reference.

    Example decisions:
    - "Primary buttons use bg-blue-600, not bg-blue-500"
    - "Card corners are rounded-xl (12px)"
    - "Error states use red-600 with shake animation"
    """
    self.decisions.append({
        "timestamp": datetime.now(),
        "category": decision.category,
        "rule": decision.rule,
        "rationale": decision.rationale,
    })
    self.persist()

def recall_relevant(self, context: str) -> list[DesignDecision]:
    """
    Retrieve decisions relevant to current context.
    """
    # Semantic search over past decisions
    return self.vector_search(context, top_k=5)

def inject_into_prompt(self, base_prompt: str) -> str:
    """
    Augment prompt with relevant past decisions.
    """
    relevant = self.recall_relevant(base_prompt)

    if not relevant:
        return base_prompt

    decisions_context = "\n".join([
        f"- {d.rule} (Rationale: {d.rationale})"
        for d in relevant
    ])

    return f"""
    {base_prompt}

    ## Past Design Decisions (maintain consistency):
    {decisions_context}
    """

Modes and Permissions

Agent Modes

Configure agent behavior for different contexts:

class UIAgentModes: """ Different operational modes for UI agents. """

MODES = {
    "draft": {
        "description": "Fast, exploratory generation",
        "quality_threshold": 0.6,
        "iterations": 1,
        "include_comments": True,
        "placeholder_content": True,
    },
    "production": {
        "description": "High-quality, deployment-ready",
        "quality_threshold": 0.9,
        "iterations": 5,
        "include_comments": False,
        "placeholder_content": False,
    },
    "audit": {
        "description": "Read-only review mode",
        "can_modify": False,
        "generate_report": True,
    },
    "learning": {
        "description": "Explain decisions, teach patterns",
        "verbose_reasoning": True,
        "cite_sources": True,
    },
}

Permission Boundaries

Define what agents can and cannot do:

class AgentPermissions: """ Capability boundaries for UI agents. """

# File system permissions
READ_CODEBASE = "read_codebase"
WRITE_COMPONENTS = "write_components"
WRITE_STYLES = "write_styles"
MODIFY_CONFIG = "modify_config"

# Tool permissions
ACCESS_BROWSER = "access_browser"
ACCESS_FIGMA = "access_figma"
RUN_TESTS = "run_tests"
DEPLOY_PREVIEW = "deploy_preview"

# Common permission sets
READONLY_REVIEWER = [READ_CODEBASE, ACCESS_BROWSER]
COMPONENT_BUILDER = [READ_CODEBASE, WRITE_COMPONENTS, WRITE_STYLES]
FULL_ACCESS = [READ_CODEBASE, WRITE_COMPONENTS, WRITE_STYLES,
               MODIFY_CONFIG, ACCESS_BROWSER, RUN_TESTS]

Anti-Patterns to Avoid

1. Monolithic Mega-Prompt

Problem: Stuffing all instructions into one giant prompt Solution: Use hierarchical delegation with focused agents

2. Context Overflow

Problem: Exceeding context window with full design system Solution: Use RAG to inject relevant context dynamically

3. No Feedback Loop

Problem: Single-pass generation with no validation Solution: Implement review-refine loops with quality thresholds

4. Hardcoded Workflows

Problem: Rigid pipelines that can't adapt Solution: Dynamic orchestration based on task complexity

5. Agent Anarchy

Problem: Too many agents with unclear responsibilities Solution: Clear separation of concerns, explicit handoff protocols

Quick Reference

Scenario Recommended Pattern

Simple component Synthesis-Master

Full page design Hierarchical Delegation

Design system work Specialized Agent Swarm

Rapid prototyping Draft mode + single agent

Production deployment Full pipeline with review

A/B testing designs Parallel Variant Generation

Integration with LibreUIUX

This skill works best when combined with:

• design-mastery/design-principles

• Feed principles to Design Agent

• archetypal-alchemy/jungian-archetypes

• Personality for UI generation

• context-management/design-system-context

• Token management

• mcp-integrations/browser-devtools-mcp

• Live inspection tools

"The agent is not the code - it is the intention made executable."