ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

LangChain for Production-Ready LLM Applications
Chapter 1: Advanced LangChain Architecture and Customization
LangChain Expression Language (LCEL) Internals
Asynchronous Operations and Concurrency
Customizing Core Components: LLMs, Prompts, Parsers
Advanced Output Parsing Strategies
Managing State in Complex Chains
Debugging LangChain Execution Flow
Hands-on Practical: Building a Custom Chain Component
Chapter 2: Building Sophisticated Agents and Tools
Agent Architectures: ReAct, Self-Ask, Plan-and-Execute
Developing Custom Tools for Agents
Handling Tool Errors and Agent Recovery
Multi-Agent Systems and Collaboration Patterns
Structured Tool Calling and Function Integration
Agent Execution Tracing and Analysis
Practice: Creating an Agent with Custom API Tools
Chapter 3: Advanced Memory Management Techniques
Comparing Advanced Memory Types
Implementing Persistent Memory Stores
Context Window Management Strategies
Custom Memory Module Development
Integrating Memory with Agents and Chains
Handling Memory in Asynchronous Applications
Hands-on Practical: Implementing Vector Store Memory
Chapter 4: Production-Grade Data Integration and Retrieval
Advanced Document Loading and Transformation
Vector Store Selection and Optimization at Scale
Advanced Indexing Strategies
Hybrid Search Implementation
Re-ranking and Query Transformation
Managing Data Updates and Synchronization
Practice: Building an Optimized RAG Pipeline
Chapter 5: Evaluation, Monitoring, and Observability
Introduction to LangSmith for Production
Defining Custom Evaluation Metrics
Automated Evaluation Pipelines
Using LangSmith for Debugging and Root Cause Analysis
Monitoring Application Performance and Cost
Integrating with Third-Party Observability Platforms
Human-in-the-Loop Feedback and Annotation
Practice: Evaluating an Agent with LangSmith
Chapter 6: Optimizing and Scaling LangChain Applications
Identifying Performance Bottlenecks
LLM Call Optimization Techniques
Cost Management and Token Usage Tracking
Scaling Data Retrieval Systems
Handling High Concurrency and Throughput
Batch Processing for Offline Tasks
Practice: Performance Tuning a LangChain Chain
Chapter 7: Deployment Strategies for Production
Structuring LangChain Projects for Deployment
Containerizing LangChain Applications with Docker
Deployment Options: Servers, Kubernetes, Serverless
Serverless Deployment Patterns for LangChain
Managing Environment Variables and Secrets
Setting Up CI/CD Pipelines
Blue/Green and Canary Deployment Strategies
Hands-on Practical: Deploying a LangChain App via Docker
Chapter 8: Security Considerations in LangChain Applications
Understanding Attack Vectors in LLM Applications
Input Validation and Sanitization
Mitigating Prompt Injection Risks
Secure Output Handling and Parsing
Securing Custom Tools and API Interactions
Data Privacy and Handling Sensitive Information
Dependency Security Management
Practice: Implementing Input Validation

Practice: Evaluating an Agent with LangSmith

Theory provides the foundation, but practical application solidifies understanding. Learn to evaluate a LangChain agent using LangSmith. This involves setting up a simple agent, creating an evaluation dataset, running the agent against the dataset, and implementing custom evaluation logic to assess its performance programmatically.

This exercise assumes you have LangSmith set up and your API key configured in your environment (LANGCHAIN_API_KEY). You should also have basic familiarity with creating LangChain agents and using tools.

1. Define the Agent Under Test

First, let's define a straightforward agent that uses a search tool. We'll use Tavily as our search tool for this example, but you could substitute another search tool or custom tools. Ensure you have the necessary packages installed (langchain, langchain_community, langchain_openai, tavily-python, langsmith). Also, set your Tavily API key (TAVILY_API_KEY) and OpenAI API key (OPENAI_API_KEY) as environment variables.

import os
from langchain_openai import ChatOpenAI
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain import hub
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langsmith import Client

# Ensure API keys are set as environment variables
# os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
# os.environ["TAVILY_API_KEY"] = "YOUR_TAVILY_API_KEY"
# os.environ["LANGCHAIN_API_KEY"] = "YOUR_LANGSMITH_API_KEY"
# os.environ["LANGCHAIN_TRACING_V2"] = "true" # Ensure tracing is enabled
# os.environ["LANGCHAIN_PROJECT"] = "Agent Evaluation Example" # Optional: Define a LangSmith project

# Initialize the LLM and Tool
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
search_tool = TavilySearchResults(max_results=2)
tools = [search_tool]

# Get the prompt template
# Using a standard OpenAI Tools Agent prompt
prompt = hub.pull("hwchase17/openai-tools-agent")

# Create the agent
# This agent is designed to work with models that support tool calling
agent = create_tool_calling_agent(llm, tools, prompt)

# Create the AgentExecutor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Test invocation (optional)
# print(agent_executor.invoke({"input": "What was the score of the last SF Giants game?"}))

We now have agent_executor, which represents the system we want to evaluate.

2. Create an Evaluation Dataset in LangSmith

Evaluation requires a set of inputs and, ideally, expected outputs or criteria against which to judge the agent's performance. Let's create a small dataset directly in LangSmith using the client library. We'll include inputs (questions for our agent) and optional reference outputs.

# Initialize LangSmith client
client = Client()

dataset_name = "Simple Search Agent Questions V1"
dataset_description = "Basic questions requiring web search."

# Check if dataset exists, create if not
try:
    dataset = client.read_dataset(dataset_name=dataset_name)
    print(f"Dataset '{dataset_name}' already exists.")
except Exception: # LangSmith client raises a generic Exception if not found
    dataset = client.create_dataset(
        dataset_name=dataset_name,
        description=dataset_description,
    )
    print(f"Created dataset '{dataset_name}'.")

    # Define examples (input questions and optional reference outputs)
    examples = [
        ("What is the capital of France?", "Paris"),
        ("Who won the 2023 Formula 1 Championship?", "Max Verstappen"),
        ("What is the main component of air?", "Nitrogen"),
        ("Summarize the plot of the movie 'Inception'.", "A thief who steals information by entering people's dreams takes on the inverse task of planting an idea into a target's subconscious."), # Example reference output
    ]

    # Add examples to the dataset
    for input_query, reference_output in examples:
        client.create_example(
            inputs={"input": input_query},
            outputs={"reference": reference_output}, # Using 'reference' key for the expected output
            dataset_id=dataset.id,
        )
    print(f"Added {len(examples)} examples to dataset '{dataset_name}'.")

After running this code, you should see a new dataset named "Simple Search Agent Questions V1" in your LangSmith account, populated with the defined examples. The outputs dictionary in create_example can store reference values, labels, or any other information useful for evaluation.

3. Run Evaluation Using LangSmith

With the agent defined and the dataset created, we can now run the agent over each example in the dataset using LangSmith's evaluation utilities. We'll start without a custom evaluator, primarily to collect traces and observe behavior.

from langsmith.evaluation import evaluate

# Define a function that encapsulates the agent invocation
# This is needed for the evaluate function
def agent_predictor(inputs: dict) -> dict:
    """Runs the agent executor for a given input dictionary."""
    return agent_executor.invoke({"input": inputs["input"]}) # Assumes dataset input is "input"

# Run the evaluation
# This will execute the agent_predictor for each example in the dataset
# Results and traces will be automatically logged to LangSmith
evaluation_results = evaluate(
    agent_predictor,
    data=dataset_name, # Can pass dataset name directly
    description="Initial evaluation run for the search agent.",
    project_name="Agent Eval Run - Simple Search", # Optional: Logs to a specific project run
    # metadata={"agent_version": "1.0"}, # Optional: Add metadata to the run
)

print("Evaluation run completed. Check LangSmith for results.")

Navigate to your LangSmith project. You should find a new evaluation run associated with the dataset. Click on it to explore:

  • Traces: Detailed logs of each agent execution for every example in the dataset. You can inspect the LLM calls, tool usage, inputs, and outputs.
  • Results Table: A summary view showing inputs, actual outputs, reference outputs (if provided), latency, and token counts.

4. Implement a Custom Evaluator

Simply running the agent and tracing is useful for debugging, but quantitative evaluation requires defining specific metrics. Let's create a custom evaluator function that checks if the agent's output contains the reference answer (case-insensitive).

from langsmith.evaluation import EvaluationResult, run_evaluator

@run_evaluator
def check_contains_reference(run, example) -> EvaluationResult:
    """
    Checks if the agent's output contains the reference answer (case-insensitive).

    Args:
        run: The LangSmith run object for the agent execution.
        example: The LangSmith example object from the dataset.

    Returns:
        An EvaluationResult with a score (1 for contains, 0 otherwise)
        and a descriptive.
    """
    agent_output = run.outputs.get("output") if run.outputs else None
    reference_output = example.outputs.get("reference") if example.outputs else None

    if agent_output is None or reference_output is None:
        # Handle cases where output or reference is missing
        score = 0
        comment = "Agent output or reference output missing."
    elif str(reference_output).lower() in str(agent_output).lower():
        score = 1 # Success: Reference found in agent output
        comment = "Reference answer found."
    else:
        score = 0 # Failure: Reference not found
        comment = f"Reference '{reference_output}' not found in output."

    return EvaluationResult(
        key="contains_reference", # Name of the metric
        score=score,              # The numeric score (0 or 1 here)
        comment=comment           # Optional qualitative feedback
    )

This function uses the @run_evaluator decorator, indicating it's designed for LangSmith evaluation. It accesses the agent's actual output (run.outputs) and the reference output stored in the dataset (example.outputs). It returns an EvaluationResult object containing a metric name and a score.

5. Run Evaluation with the Custom Evaluator

Now, let's re-run the evaluation, this time including our custom evaluator.

# Run evaluation again, now with the custom evaluator
custom_eval_results = evaluate(
    agent_predictor,
    data=dataset_name,
    evaluators=[check_contains_reference], # Pass the custom evaluator function
    description="Evaluation run with custom 'contains_reference' check.",
    project_name="Agent Eval Run - Custom Check", # Log to a different run project
    # metadata={"agent_version": "1.0", "evaluator": "contains_reference_v1"},
)

print("Evaluation run with custom evaluator completed. Check LangSmith.")

Go back to LangSmith and view this new evaluation run. In the results table, you should now see a new column titled contains_reference (matching the reference in our EvaluationResult). This column will display the score (0 or 1) for each example based on our custom logic. You can sort and filter by this metric to quickly identify failures. Hovering over or clicking into the feedback cell often shows the comment provided by the evaluator.

If we were to visualize the results of this simple evaluation (hypothetically, based on the contains_reference scores), it might look something like this:

A simple bar chart showing the count of examples passing (score=1) and failing (score=0) the contains_reference evaluation metric.

Further Steps

This practical exercise demonstrated the core loop of evaluating an agent with LangSmith: defining the agent, creating a dataset, running evaluation, and implementing custom checks. From here, you can build more complex evaluations:

  • More Sophisticated Evaluators: Implement evaluators using regular expressions, semantic similarity checks (e.g., comparing embeddings of actual and reference outputs), or checks for specific structural properties of the output.
  • LLM-as-Judge: Use another LLM to evaluate the agent's output based on criteria like helpfulness, correctness (against the reference), or lack of harmful content. LangChain provides helpers for creating these evaluators.
  • Multiple Evaluators: Pass a list of different evaluator functions to evaluate to calculate multiple metrics simultaneously.
  • Dataset Management: Develop strategies for curating and versioning your evaluation datasets as your application evolves.
  • Integration with CI/CD: Automate these evaluation runs as part of your continuous integration pipeline to catch regressions before deployment.

Systematic evaluation using tools like LangSmith is indispensable for building reliable LLM applications. It goes further than anecdotal testing, providing quantifiable metrics and detailed tracing to understand and improve agent performance over time.

© 2026 ApX Machine Learning