langsmith — LLM Observability, Evaluation & Prompt Management

Keyword: langsmith · llm tracing · llm evaluation · @traceable · langsmith evaluate

LangSmith is a framework-agnostic platform for developing, debugging, and deploying LLM applications. It provides end-to-end tracing, quality evaluation, prompt versioning, and production monitoring.

When to use this skill

• Add tracing to any LLM pipeline (OpenAI, Anthropic, LangChain, custom models)

• Run offline evaluations with evaluate() against a curated dataset

• Set up production monitoring and online evaluation

• Manage and version prompts in the Prompt Hub

• Create datasets for regression testing and benchmarking

• Attach human or automated feedback to traces

• Use LLM-as-judge scoring with openevals

• Debug agent failures with end-to-end trace inspection

Instructions

• Install SDK: pip install -U langsmith (Python) or npm install langsmith (TypeScript)

• Set environment variables: LANGSMITH_TRACING=true , LANGSMITH_API_KEY=lsv2_...

• Instrument with @traceable decorator or wrap_openai() wrapper

• View traces at smith.langchain.com

• For evaluation setup, see references/python-sdk.md

• For CLI commands, see references/cli.md

• Run bash scripts/setup.sh to auto-configure environment

API Key: Get from smith.langchain.com → Settings → API Keys Docs: https://docs.langchain.com/langsmith

Quick Start

Python

pip install -U langsmith openai export LANGSMITH_TRACING=true export LANGSMITH_API_KEY="lsv2_..." export OPENAI_API_KEY="sk-..."

from langsmith import traceable from langsmith.wrappers import wrap_openai from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable def rag_pipeline(question: str) -> str: """Automatically traced in LangSmith""" response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": question}] ) return response.choices[0].message.content

result = rag_pipeline("What is LangSmith?")

TypeScript

npm install langsmith openai export LANGSMITH_TRACING=true export LANGSMITH_API_KEY="lsv2_..."

import { traceable } from "langsmith/traceable"; import { wrapOpenAI } from "langsmith/wrappers"; import { OpenAI } from "openai";

const client = wrapOpenAI(new OpenAI());

const pipeline = traceable(async (question: string): Promise<string> => { const res = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: question }], }); return res.choices[0].message.content ?? ""; }, { name: "RAG Pipeline" });

await pipeline("What is LangSmith?");

Core Concepts

Concept Description

Run Individual operation (LLM call, tool call, retrieval). The fundamental unit.

Trace All runs from a single user request, linked by trace_id .

Thread Multiple traces in a conversation, linked by session_id or thread_id .

Project Container grouping related traces (set via LANGSMITH_PROJECT ).

Dataset Collection of {inputs, outputs} examples for offline evaluation.

Experiment Result set from running evaluate() against a dataset.

Feedback Score/label attached to a run — numeric, categorical, or freeform.

Tracing

@traceable decorator (Python)

from langsmith import traceable

@traceable( run_type="chain", # llm | chain | tool | retriever | embedding name="My Pipeline", tags=["production", "v2"], metadata={"version": "2.1", "env": "prod"}, project_name="my-project" ) def pipeline(question: str) -> str: return generate_answer(question)

Selective tracing context

import langsmith as ls

Enable tracing for this block only

with ls.tracing_context(enabled=True, project_name="debug"): result = chain.invoke({"input": "..."})

Disable tracing despite LANGSMITH_TRACING=true

with ls.tracing_context(enabled=False): result = chain.invoke({"input": "..."})

Wrap provider clients

from langsmith.wrappers import wrap_openai, wrap_anthropic from openai import OpenAI import anthropic

openai_client = wrap_openai(OpenAI()) # All calls auto-traced anthropic_client = wrap_anthropic(anthropic.Anthropic())

Distributed tracing (microservices)

from langsmith.run_helpers import get_current_run_tree import langsmith

@langsmith.traceable def service_a(inputs): rt = get_current_run_tree() headers = rt.to_headers() # Pass to child service return call_service_b(headers=headers)

@langsmith.traceable def service_b(x, headers): with langsmith.tracing_context(parent=headers): return process(x)

Evaluation

Basic evaluation with evaluate()

from langsmith import Client from langsmith.wrappers import wrap_openai from openai import OpenAI

client = Client() oai = wrap_openai(OpenAI())

1. Create dataset

dataset = client.create_dataset("Geography QA") client.create_examples( dataset_id=dataset.id, examples=[ {"inputs": {"q": "Capital of France?"}, "outputs": {"a": "Paris"}}, {"inputs": {"q": "Capital of Germany?"}, "outputs": {"a": "Berlin"}}, ] )

2. Target function

def target(inputs: dict) -> dict: res = oai.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": inputs["q"]}] ) return {"a": res.choices[0].message.content}

3. Evaluator

def exact_match(inputs, outputs, reference_outputs): return outputs["a"].strip().lower() == reference_outputs["a"].strip().lower()

4. Run experiment

results = client.evaluate( target, data="Geography QA", evaluators=[exact_match], experiment_prefix="gpt-4o-mini-v1", max_concurrency=4 )

LLM-as-judge with openevals

pip install -U openevals

from openevals.llm import create_llm_as_judge from openevals.prompts import CORRECTNESS_PROMPT

judge = create_llm_as_judge( prompt=CORRECTNESS_PROMPT, model="openai:o3-mini", feedback_key="correctness", )

results = client.evaluate(target, data="my-dataset", evaluators=[judge])

Evaluation types

Type When to use

Code/Heuristic Exact match, format checks, rule-based

LLM-as-judge Subjective quality, safety, reference-free

Human Annotation queues, pairwise comparison

Pairwise Compare two app versions

Online Production traces, real traffic

Prompt Hub

from langsmith import Client from langchain_core.prompts import ChatPromptTemplate

client = Client()

Push a prompt

prompt = ChatPromptTemplate([ ("system", "You are a helpful assistant."), ("user", "{question}"), ]) client.push_prompt("my-assistant-prompt", object=prompt)

Pull and use

prompt = client.pull_prompt("my-assistant-prompt")

Pull specific version:

prompt = client.pull_prompt("my-assistant-prompt:abc123")

Feedback

from langsmith import Client import uuid

client = Client()

Custom run ID for later feedback linking

my_run_id = str(uuid.uuid4()) result = chain.invoke({"input": "..."}, {"run_id": my_run_id})

Attach feedback

client.create_feedback( key="correctness", score=1, # 0-1 numeric or categorical run_id=my_run_id, comment="Accurate and concise" )

References

• Python SDK Reference — full Client API, @traceable signature, evaluate()

• TypeScript SDK Reference — Client, traceable, wrappers, evaluate

• CLI Reference — langsmith CLI commands

• Official Docs — langchain.com/langsmith

• SDK GitHub — MIT License, v0.7.17

• openevals — Prebuilt LLM evaluators