OpenTester Testing Skill

MCP-First testing execution infrastructure for AI coding tools.

Prerequisites

1. Install OpenTester:

   pip install opentester
   

2. Start OpenTester server:

   opentester start
   

3. Services:

   • Web UI: http://localhost:8000
   • MCP Server: http://localhost:8001/mcp

Quick Start

Available Commands

MCP Tools

Template Tools

DSL Validation Workflow (CRITICAL)

ALWAYS validate DSL before saving!

1. Generate DSL YAML
2. Call validate_dsl(dsl_yaml) to check syntax
3. IF validation fails:
   - Read the error message
   - Fix the DSL according to error
   - Call validate_dsl again
   - REPEAT until validation passes
4. IF validation passes:
   - Call save_case to store the test
5. Call run_case to execute

Validation Error Handling

Common errors and fixes:

DSL Format (Correct)

Basic Structure

version: "1.0"
meta:
  name: "Test Name"
  description: "What this test verifies"
  author: "Author"           # Optional
  tags: ["tag1", "tag2"]     # Optional

target:
  type: cli                  # cli, api, or tui

timeout: 30                  # Optional: Default timeout (seconds)

vars:                        # Optional: Variables
  base_url: "http://localhost:3000"

steps:
  - name: "Step description"
    action: exec|assert|wait
    # Action-specific fields below

Action: exec (Execute Command)

CLI Target:

- name: "Execute shell command"
  action: exec
  command: "echo Hello World"     # Required: Command to execute
  working_dir: "/path/to/dir"     # Optional: Working directory
  env:                            # Optional: Environment variables
    KEY: "value"
  timeout: 10                     # Optional: Timeout in seconds

API Target:

- name: "Send HTTP request"
  action: exec
  method: GET                     # HTTP method: GET, POST, PUT, DELETE
  path: "/api/endpoint"           # API path (appended to base_url)
  body:                           # Optional: Request body
    key: "value"
  headers:                        # Optional: Additional headers
    X-Custom: "value"

TUI Target:

- name: "Send input to TUI"
  action: exec
  input: "text to type"           # Text input
  # OR
  key: "enter"                    # Special key: enter, tab, escape, etc.

Action: assert (Verify Conditions)

⚠️ CRITICAL: assert MUST use assertion field, NOT type directly!

CLI Assertions:

- name: "Verify exit code"
  action: assert
  assertion:
    type: exit_code               # Assertion type
    expected: 0                   # Expected value (use 'expected', not 'value'!)

- name: "Verify stdout contains text"
  action: assert
  assertion:
    type: stdout_contains
    expected: "expected text"

- name: "Verify stderr contains error"
  action: assert
  assertion:
    type: stderr_contains
    expected: "error message"

- name: "Verify stdout matches pattern"
  action: assert
  assertion:
    type: stdout_matches
    expected: "^pattern$"

- name: "Verify stderr is empty"
  action: assert
  assertion:
    type: stderr_empty
    expected: true

API Assertions:

- name: "Verify HTTP status"
  action: assert
  assertion:
    type: status_code
    expected: 200

- name: "Verify JSON response"
  action: assert
  assertion:
    type: json_path
    path: "$.data.name"           # JSON path
    expected: "expected value"

TUI Assertions:

- name: "Verify screen content"
  action: assert
  assertion:
    type: screen_contains
    expected: "visible text"

- name: "Verify cursor position"
  action: assert
  assertion:
    type: cursor_at
    expected: "prompt"

Action: wait (Pause Execution)

- name: "Wait for process"
  action: wait
  duration: 5                     # Seconds to wait

Error Handling

- name: "Optional step that may fail"
  action: exec
  command: "optional-command"
  on_error: continue              # Options: fail (default), continue, stop

Target Types

CLI Target

target:
  type: cli
  working_dir: "/path/to/dir"     # Optional
  env:                            # Optional
    KEY: "value"

API Target

target:
  type: api
  base_url: "http://api.example.com"  # Required
  headers:                            # Optional default headers
    Authorization: "Bearer token"

TUI Target

target:
  type: tui
  start_command: "my-tui-app"     # Required: Command to start TUI
  terminal:                       # Optional
    rows: 24
    cols: 80

Variables

vars:
  base_url: "http://localhost:3000"
  timeout: 30

steps:
  - name: "Use variable"
    action: exec
    command: "curl ${vars.base_url}/api"
    timeout: "${vars.timeout}"

Complete Example (CLI)

version: "1.0"
meta:
  name: "Echo Command Test"
  description: "Test basic echo functionality"

target:
  type: cli

timeout: 10

steps:
  - name: "Execute echo command"
    action: exec
    command: echo "Hello World"

  - name: "Verify exit code is 0"
    action: assert
    assertion:
      type: exit_code
      expected: 0

  - name: "Verify output contains expected text"
    action: assert
    assertion:
      type: stdout_contains
      expected: "Hello World"

Complete Example (API)

version: "1.0"
meta:
  name: "API Health Check"
  description: "Test API health endpoint"

target:
  type: api
  base_url: "http://localhost:8000"

timeout: 10

steps:
  - name: "Call health endpoint"
    action: exec
    method: GET
    path: "/health"

  - name: "Verify status code"
    action: assert
    assertion:
      type: status_code
      expected: 200

  - name: "Verify response body"
    action: assert
    assertion:
      type: json_path
      path: "$.status"
      expected: "ok"

Workflows

Create and Run Tests Workflow

When user asks to create tests:

1. Discover: Call list_projects to find existing projects
2. Context: Read relevant source files and understand changes
3. Generate: Create DSL YAML following the CORRECT format above
4. Validate: Call validate_dsl(dsl_yaml)
5. Check Result:
   • If ❌ validation fails:
     • Read error message carefully
     • Fix the DSL (usually: change type/value to assertion: { type, expected })
     • Call validate_dsl again
     • Repeat until ✅ validation passes
6. Save: Call save_case with validated DSL
7. Execute: Call run_case to run tests
8. Report: Show execution results

Delete Test Case Workflow

When user asks to delete a test case:

1. Find: Call get_project(project_id) to confirm project and case exist
2. Confirm: Show user the case details and ask for confirmation
3. Delete: Call delete_case(project_id, case_id)
4. Verify: Call get_project(project_id) to confirm deletion
5. Report: Show deletion result

User: "Delete test case 'login-test' from project 'auth-module'"

Agent:
1. Get project details to find the case
2. Confirm: "Delete test case 'login-test' (ID: xxx)?"
3. delete_case(project_id="auth-module", case_id="xxx")
4. Report: "✅ Test case 'login-test' deleted successfully"

Delete Project Workflow

When user asks to delete a project:

1. Warn: Show warning that this will delete ALL test cases in the project
2. List: Call get_project(project_id) to show what will be deleted
3. Confirm: Ask user to confirm the deletion
4. Delete: Call delete_project(project_id)
5. Report: Show deletion result

User: "Delete project 'legacy-tests'"

Agent:
1. Get project details
2. Warn: "⚠️ This will permanently delete project 'legacy-tests' with 12 test cases"
3. Confirm: "Are you sure? (yes/no)"
4. delete_project(project_id="legacy-tests")
5. Report: "✅ Project 'legacy-tests' and all its test cases deleted"

Cleanup Old Executions Workflow

When user wants to clean up old test data:

1. List: Call list_projects to show all projects
2. Identify: Ask user which projects/test cases to delete
3. Delete: Follow delete workflow above
4. Report: Summary of cleanup actions

Template Workflow - List and Use Templates

When user wants to use a template:

1. List: Call list_templates to show available templates
2. Filter (optional): Use category, target_type, or tag parameters
3. Get Details: Call get_template(template_id) to see variable requirements
4. Collect Variables: Ask user for variable values
5. Instantiate: Call instantiate_template with variables
6. Report: Show created test case

User: "Use the API health check template"

Agent:
1. list_templates(category="api")
2. get_template(template_id="api-health-check")
3. "Template requires: base_url (default: http://localhost:8000)"
4. instantiate_template(
     template_id="api-health-check",
     project_id="my-project",
     case_name="My API Health Test",
     variables={"base_url": "http://localhost:3000"}
   )
5. Report: "✅ Test case created from template"

Template Workflow - Create Template from Case

When user wants to save a test case as a template:

1. Get Project: Call get_project(project_id) to find the case
2. Identify Variables: Ask user which values should become variables
3. Create: Call create_template_from_case with variable mappings
4. Report: Show created template

User: "Save 'login-test' as a template"

Agent:
1. get_project(project_id="auth-module")
2. "Which values should become variables?"
   - User: "http://localhost:3000 -> base_url"
   - User: "test@example.com -> email"
3. create_template_from_case(
     project_id="auth-module",
     case_id="login-test",
     template_name="User Login Template",
     variable_mappings={
       "http://localhost:3000": "base_url",
       "test@example.com": "email"
     }
   )
4. Report: "✅ Template created with 2 variables"

Template Workflow - Create New Template

When user wants to create a template from scratch:

1. Define: Ask for template name, description, category
2. Create DSL: Create DSL with ${variable} placeholders
3. Create: Call create_template with the DSL
4. Report: Show created template

User: "Create a template for database backup tests"

Agent:
1. "Template name? Category?"
2. Create DSL with variables like ${db_name}, ${backup_path}
3. create_template(
     name="Database Backup Test",
     dsl_template="version:... steps:...",
     category="database"
   )
4. Report: "✅ Template created with 2 variables"

Template DSL Format

Templates use ${variable_name} placeholders that get replaced when instantiated.

version: "1.0"
meta:
  name: "${test_name}"
  description: "API test for ${endpoint}"

target:
  type: api
  base_url: "${base_url}"

steps:
  - name: "Call ${endpoint}"
    action: exec
    method: GET
    path: "${endpoint}"

  - name: "Verify status"
    action: assert
    assertion:
      type: status_code
      expected: ${expected_status}

When instantiating, provide variable values:

instantiate_template(
    template_id="...",
    project_id="...",
    case_name="My Test",
    variables={
        "test_name": "User API Test",
        "endpoint": "/api/users",
        "base_url": "http://localhost:8000",
        "expected_status": 200
    }
)

Common Mistakes to Avoid

❌ WRONG - assert with direct type/value:

- action: assert
  type: stdout_contains      # ❌ Wrong!
  value: "text"              # ❌ Wrong!

✅ CORRECT - assert with assertion object:

- action: assert
  assertion:                 # ✅ Correct!
    type: stdout_contains
    expected: "text"         # ✅ Use 'expected', not 'value'!

❌ WRONG - Missing name field:

- action: exec
  command: "echo test"

✅ CORRECT - With name:

- name: "Execute test command"  # ✅ Always add name
  action: exec
  command: "echo test"