Structuring Complex Tool Outputs for LLMs

When your Python tool completes its task, how it presents the results back to the Large Language Model (LLM) is just as significant as the task itself. An LLM, while proficient with natural language, thrives on predictability and structure when consuming data from external sources. Simply returning a block of undifferentiated text can lead to misinterpretations, require more complex prompting to parse, or even cause the LLM to overlook important details. Structuring your tool's output effectively is fundamental for building reliable and efficient agentic systems.

The primary goal is to make the LLM's job of understanding and utilizing the tool's output as straightforward as possible. This usually means providing data in a format that is easy for a machine to parse and for the LLM to "reason" about.

Choosing an Output Format

While tools can technically output plain text, and LLMs can often make sense of it, structured formats are superior for consistency and reliability.

JSON (JavaScript Object Notation): This is the most widely recommended and used format for tool outputs intended for LLMs.
- It's lightweight and human-readable.
- Almost all programming languages, including Python, have excellent built-in support for parsing and generating JSON.
- Its key-value pair structure, along with support for arrays and nested objects, allows for rich, hierarchical data representation.
- LLMs are generally well-trained to understand and extract information from JSON. Many LLM agent frameworks and function/tool-calling APIs explicitly require or work best with JSON.
XML (Extensible Markup Language): If your tool interacts with older systems or APIs that exclusively provide XML, you might need to return XML or convert it. However, if you have a choice, JSON is generally preferred for new tool development due to its simplicity and more common usage in modern web APIs.
Plain Text (with caveats): For extremely simple outputs where the structure is trivial (e.g., a tool that just returns a single number or a boolean status), plain text might suffice. However, as soon as there are multiple pieces of information, or if the information has inherent relationships, JSON becomes a better choice. If you must use plain text for complex data, establish a very clear, consistent, and simple-to-parse delimiter-based system, but be aware this is more prone to LLM misinterpretation than JSON.

For the majority of custom Python tools, JSON is the recommended output format.

Designing Effective Output Schemas

The "schema" refers to the blueprint of your structured output. A well-designed schema ensures the LLM receives data in a predictable and useful manner.

Clarity and Descriptiveness

Field names (keys in JSON) should be self-explanatory. Avoid cryptic abbreviations. If a field represents a user's email address, user_email or email_address is much better than u_em or dat1. Remember, the LLM uses these names to understand the meaning of the values.

Consistency

Maintain consistency in naming conventions (e.g., snake_case or camelCase for keys) and data types for similar pieces of information across different tools or even within the same tool's various outputs. If a user ID is an integer in one part of your system, it should be an integer everywhere.

Data Types

Use appropriate JSON data types:

Strings: For textual data.
Numbers: For integers and floating-point values.
Booleans: For true/false states.
Arrays: For lists of items (e.g., a list of product names, search results).
Objects: For grouping related key-value pairs (nesting).
Null: To explicitly indicate the absence of a value for a field.

Balancing Detail and Conciseness

Include all the information the LLM is likely to need for its current task or subsequent steps. However, be mindful of verbosity. Overly large outputs can consume valuable tokens in the LLM's context window and might be harder for the LLM to process efficiently. If a tool generates a massive amount of data, examine whether the tool itself can perform some summarization or filtering, or if pagination is appropriate.

Representing Errors

If a tool encounters an error it can handle gracefully, or if it needs to convey a failure state, the output structure should accommodate this. Instead of raising an exception that halts the agent (unless that's the desired behavior), the tool can return a JSON object indicating success or failure.

For example, a successful output might look like:

{
  "status": "success",
  "data": {
    "user_id": "12345",
    "username": "llm_dev"
  }
}

And an error output:

{
  "status": "error",
  "error_code": "USER_NOT_FOUND",
  "message": "User with ID '67890' could not be found."
}

This allows the LLM to understand that the tool executed but encountered a specific issue, which it can then use in its reasoning.

Examples of Structured Outputs

Let's examine a tool that fetches product information.

A less ideal, somewhat unstructured text output:

Product: SuperWidget, Price: $29.99, In Stock: Yes, Features: Durable, Lightweight, Color: Blue.

An LLM might parse this, but it's less reliable. What if "In Stock" was "No"? Or the currency symbol changed?

A well-structured JSON output for the same product:

{
  "product_name": "SuperWidget",
  "price": 29.99,
  "currency": "USD",
  "in_stock": true,
  "features": ["Durable", "Lightweight"],
  "specifications": {
    "color": "Blue",
    "weight_grams": 150
  }
}

This JSON output is unambiguous. The LLM can easily extract the price, check stock status, or list features.

For a tool returning multiple items, like search results:

{
  "query_terms": ["python", "async", "web server"],
  "results_count": 2,
  "items": [
    {
      "title": "Building Async Web Servers with Python",
      "url": "https://example.com/async-servers",
      "snippet": "An in-depth guide to using asyncio and libraries like aiohttp...",
      "relevance_score": 0.85
    },
    {
      "title": "Understanding Python's Async/Await",
      "url": "https://another-site.org/python-async",
      "snippet": "A foundational explanation of asynchronous programming concepts in Python.",
      "relevance_score": 0.72
    }
  ]
}

Here, the structure clearly delineates each search result as an object within an array, each with its own set of attributes.

Implementing Structured Outputs in Python

Python's standard library makes generating JSON straightforward. The json module is your primary utility.

import json

def get_product_info_tool(product_id: str):
    # Imagine fetching data from a database or API
    if product_id == "SW001":
        product_data = {
            "product_name": "SuperWidget",
            "price": 29.99,
            "currency": "USD",
            "in_stock": True, # Python boolean
            "features": ["Durable", "Lightweight"],
            "specifications": {
                "color": "Blue",
                "weight_grams": 150
            }
        }
        # Serialize the Python dictionary to a JSON string
        return json.dumps(product_data)
    else:
        error_data = {
            "status": "error",
            "error_code": "PRODUCT_NOT_FOUND",
            "message": f"Product with ID '{product_id}' not found."
        }
        return json.dumps(error_data)

# Example usage:
output_json = get_product_info_tool("SW001")
print(output_json)
# Output:
# {"product_name": "SuperWidget", "price": 29.99, "currency": "USD", "in_stock": true, "features": ["Durable", "Lightweight"], "specifications": {"color": "Blue", "weight_grams": 150}}

output_error_json = get_product_info_tool("XYZ789")
print(output_error_json)
# Output:
# {"status": "error", "error_code": "PRODUCT_NOT_FOUND", "message": "Product with ID 'XYZ789' not found."}

Notice how Python True is converted to JSON true. The json.dumps() function handles these translations.

For more complex scenarios, especially when dealing with many different output structures or requiring validation of your output data before serialization, libraries like Pydantic are invaluable. Pydantic allows you to define data models using Python type hints. These models can then parse input data and serialize to JSON, ensuring your output conforms to a predefined schema.

from typing import List, Dict, Optional
from pydantic import BaseModel
import json

# Define Pydantic models for our product structure
class ProductSpecifications(BaseModel):
    color: str
    weight_grams: int

class ProductInfo(BaseModel):
    product_name: str
    price: float
    currency: str
    in_stock: bool
    features: List[str]
    specifications: ProductSpecifications

class ErrorResponse(BaseModel):
    status: str = "error"
    error_code: str
    message: str

def get_product_info_tool_pydantic(product_id: str) -> str:
    if product_id == "SW001":
        product_data = ProductInfo(
            product_name="SuperWidget",
            price=29.99,
            currency="USD",
            in_stock=True,
            features=["Durable", "Lightweight"],
            specifications=ProductSpecifications(color="Blue", weight_grams=150)
        )
        # Pydantic models have a .model_dump_json() method (or .json() in Pydantic v1)
        return product_data.model_dump_json()
    else:
        error_data = ErrorResponse(
            error_code="PRODUCT_NOT_FOUND",
            message=f"Product with ID '{product_id}' not found."
        )
        return error_data.model_dump_json()

# print(get_product_info_tool_pydantic("SW001"))

Using Pydantic adds a layer of robustness by validating that the data you're about to serialize matches the expected structure and types.

The following diagram illustrates the typical flow of data from your Python tool to the LLM when using structured outputs:

Data flow from tool execution to LLM consumption when using structured JSON output.

Guiding the LLM to Use Structured Output

Simply returning JSON isn't always enough. The LLM needs to know what to expect and how to use it. This is typically achieved through:

Tool Descriptions: When you define your tool for the LLM agent (as discussed in Chapter 1, "Foundations of LLM Agent Tooling"), you describe its purpose, inputs, AND its output format. Clearly stating "This tool returns a JSON object containing X, Y, and Z fields" primes the LLM.
Function/Tool Calling Schemas: Modern LLM APIs (like OpenAI's) have explicit "function calling" or "tool calling" features. Here, you provide a JSON schema definition for your tool's input arguments and sometimes also for its output. The LLM then tries to generate arguments matching your input schema and can be better prepared to parse an output that it knows should conform to a certain structure.
Prompt Engineering: In your prompts that instruct the LLM to use tools, you can reiterate the expected output structure or give examples.

Handling Large or Complex Outputs

If your tool potentially returns a very large amount of data:

Summarization/Filtering within the Tool: Can the tool itself intelligently summarize or filter the data before returning it? For example, instead of returning 1000 search results, perhaps return the top 10 most relevant, or a summary of findings.
Indicate Truncation or Pagination: If you must truncate, make it clear in the output (e.g., "results_truncated": true, "total_available_results": 1000). If your tool supports pagination, the output could include information on how to fetch the next page (e.g., next_page_token: "nextToken123").
Streaming (Advanced): For very large datasets or continuous data, you might use streaming outputs, though this adds complexity to both the tool and the agent's handling of it.

By thoughtfully structuring your tool outputs, you significantly enhance the reliability and capability of your LLM agents. Clear, predictable JSON outputs allow the LLM to work more like a precise programmatic component rather than just a text processor, leading to more powerful and dependable applications. As you continue to develop custom Python tools, always think about how the LLM will best consume the information your tool provides.

Was this section helpful?

References

Function calling, OpenAI, 2024 - This guide details how to integrate custom tools with OpenAI's models, illustrating the definition of tool schemas and the consumption of structured tool outputs by large language models.
Pydantic Documentation, Samuel Colvin, 2024 - Official documentation for Pydantic, a Python library that aids in defining, validating, and serializing data schemas, which is particularly useful for robustly structuring tool outputs.
RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format, T. Bray, 2017 (IETF) DOI: https://doi.org/10.17487/RFC8259 - The official specification defining the JSON format, providing a common standard for representing structured data, which is fundamental for consistent tool outputs.