OpenResponses Compliance Test

Run the official OpenResponses compliance test suite against the local (or remote) Response API endpoint.

Quick Start

From the openapi package directory

cd lobehub/packages/openapi

Run all tests (dev mode, localhost:3010)

APP_URL=http://localhost:3010 bun run test:response-compliance --
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1

Run specific tests only

APP_URL=http://localhost:3010 bun run test:response-compliance --
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1
--filter basic-response,streaming-response

Verbose mode (shows request/response details)

APP_URL=http://localhost:3010 bun run test:response-compliance --
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 -v

JSON output (for CI)

APP_URL=http://localhost:3010 bun run test:response-compliance --
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 --json

Prerequisites

• Dev server running with ENABLE_MOCK_DEV_USER=true in .env

• The api/v1/responses route registered (via src/app/(backend)/api/v1/[[...route]]/route.ts )

Auth Modes

Mode Flags

Dev (mock user) --auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1

API Key --api-key lb-xxxxxxxxxxxxxxxx

Custom --auth-header <name> --api-key <value>

Test IDs

Available --filter values:

ID Description Related Issue

basic-response

Simple text generation (non-streaming) LOBE-5858

streaming-response

SSE streaming lifecycle + events LOBE-5859

system-prompt

System role message handling LOBE-5858

tool-calling

Function tool definition + call output LOBE-5860

image-input

Multimodal image URL content —

multi-turn

Conversation history via input items LOBE-5861

Environment Variables

Variable Default Description

APP_URL

http://localhost:3010

Server base URL (auto-appends /api/v1 )

API_KEY

— API key (alternative to --api-key flag)

How It Works

The script (lobehub/packages/openapi/scripts/compliance-test.sh ) clones the official openresponses/openresponses repo into scripts/openresponses-compliance/ (gitignored) and runs its CLI test runner. First run clones; subsequent runs update from upstream.

Debugging Failures

• Run with -v to see full request/response payloads

• Common failure patterns:

• "Failed to parse JSON": Auth failed, server returned HTML redirect

• "Response has no output items": LLM execution not yet implemented

• "Expected number, received null": Missing required field in response schema

• "Invalid input": Zod validation on response schema — check field format

Key Files

• Types: lobehub/packages/openapi/src/types/responses.type.ts

• Service: lobehub/packages/openapi/src/services/responses.service.ts

• Controller: lobehub/packages/openapi/src/controllers/responses.controller.ts

• Route: lobehub/packages/openapi/src/routes/responses.route.ts

• Test script: lobehub/packages/openapi/scripts/compliance-test.sh

• Cloud route: src/app/(backend)/api/v1/[[...route]]/route.ts