Prompting for Long-Term Information Retrieval

While an agent's short-term memory, operating within its context window, is useful for immediate tasks, it's quickly exhausted when dealing with substantial information or interactions over time. To perform effectively, agents must often access and utilize information from larger, more persistent knowledge repositories. This is where prompting for long-term information retrieval becomes essential. Your role as a prompt engineer is to craft instructions that enable the agent to intelligently decide when to search, what to search for, and how to use the information it finds.

Think of the agent as a skilled researcher who needs clear directives. Without them, it might not know a vast library of information is available or how to navigate it. Prompts provide these directives, bridging the agent's LLM core with external data sources.

Guiding the Retrieval Process

Effectively prompting for long-term information retrieval involves several interconnected strategies. You're not just telling the agent that a knowledge base exists; you're guiding its interaction with it.

1. Triggering the Retrieval Action

An agent needs to understand when to look for external information. Your prompt should define these triggers. This could be:

• Implicitly: By instructing the agent to always seek external information if its internal knowledge is insufficient.

  If you cannot answer the user's question with high confidence based on the current conversation or your general knowledge, you MUST use the `search_knowledge_base` tool to find relevant information before responding.
  

• Explicitly based on keywords or intent:

  If the user asks for specific data, company policies, or detailed project updates, use the `document_retrieval_tool` to find the answer.
  

• Based on uncertainty:

  If you are unsure about a factual detail or need to verify information, state your uncertainty and then use the `enterprise_search_tool` to find supporting evidence.
  

2. Crafting Effective Search Queries

Once the agent decides to retrieve information, it needs to formulate a good query. Poor queries lead to poor results. Your prompts should guide the agent in generating queries that are:

• Concise and Relevant: Instruct the agent to extract key terms or entities from the user's request.

  User asks: "What were the marketing strategies for Project Titan in Q2 and how did they impact sales figures according to the quarterly report?"
  
  Your thought process:
  1. The user needs information on "marketing strategies for Project Titan Q2" and "sales figures impact".
  2. I should query the knowledge base for "Project Titan Q2 marketing report" and "Project Titan Q2 sales results".
  
  Formulate a query for `search_documents_tool`:
  search_documents_tool("Project Titan Q2 marketing and sales report")
  

• Formatted for the Tool: If the knowledge store is a specific database or API, the query might need a particular structure.

  To query the product database, generate a SQL-like query string for the `query_product_db` tool. Example: `query_product_db("SELECT name, price FROM products WHERE category = 'electronics' AND stock > 0")`
  

• Iterative: Encourage the agent to refine queries if initial results are not satisfactory. This is part of the "Refining Instructions for Complex Agent Actions" discussed in Chapter 2, but applied to retrieval.

  If your initial search with `search_tool` does not yield relevant results, try rephrasing your query with more specific keywords or broader terms based on the context. Explain your reasoning for the new query.
  

3. Processing and Integrating Retrieved Information

Getting data back from the knowledge store is only half the battle. The agent must then process this information and integrate it coherently into its response. Prompts should guide this synthesis:

• Summarization and Extraction: If the retrieved data is lengthy, instruct the agent to summarize it or extract only the most relevant parts.

  Context: [Retrieved document text: "The Q3 financial report indicates a 15% growth in SaaS revenue, primarily driven by enterprise client acquisitions. Key challenges included increased operational costs..."]
  User Question: "What was our SaaS revenue growth in Q3?"
  
  Instruction: From the retrieved text, extract the specific percentage for SaaS revenue growth and any primary drivers mentioned.
  

• Answering the Original Question: Ensure the agent uses the retrieved information to directly address the user's query, rather than just presenting the raw data.

  After retrieving information using `fetch_policy_document`, use the content of the document to directly answer the user's question about the remote work policy. Do not just paste the document.
  

• Handling Multiple Pieces of Information: If the agent retrieves several pieces of information, or information from multiple queries, it needs to synthesize them.

  You have retrieved the following information:
  Document A: "Project Alpha is on schedule."
  Document B: "Key milestone for Project Alpha is delayed by 1 week due to unforeseen circumstances."
  
  Synthesize this information. Acknowledge any discrepancies if they exist.
  

4. Managing Imperfect or Missing Information

Retrieval systems aren't perfect. Agents need to be prepared for:

• Irrelevant Information: Instruct the agent to identify and discard clearly irrelevant search results.
• No Information Found: Provide a strategy for when the knowledge base doesn't contain the answer.

  If the `knowledge_search_tool` returns no relevant results after two attempts with refined queries, inform the user that the information could not be found in the available documents and ask if they would like you to search the public web or if they can provide more context.
  

• Conflicting Information: As seen in the example above, prompts can guide agents on how to present or question conflicting data.

Visualizing the Prompt-Guided Retrieval Flow

The following diagram illustrates how prompts guide an agent through the process of long-term information retrieval. The system prompt contains the overarching instructions, including when to retrieve, how to form queries, and how to synthesize results.

The agent, guided by the system prompt, processes a user request, determines the need for external information, formulates a query, accesses the knowledge store, and then integrates the retrieved data to generate a comprehensive final response.

Example: Prompting a Customer Support Agent for Policy Information

Let's imagine a customer support agent designed to answer questions based on a company's internal policy documents, stored in a searchable database accessible via a get_policy_info tool.

Core Instruction in the System Prompt:

You are a helpful customer support AI. Your primary goal is to answer customer questions accurately based on official company policies.
When a customer asks a question that likely pertains to a specific policy (e.g., returns, warranty, privacy, terms of service):
1.  Identify the core topic of the policy question.
2.  Formulate a concise search query for the `get_policy_info` tool. The query should be 2-5 keywords that best describe the policy topic.
    Example user question: "How long do I have to return a product I bought online?"
    Example query for `get_policy_info`: "return policy online purchase window"
3.  Execute the `get_policy_info` tool with your query.
4.  Once you receive the policy information, carefully review it.
5.  Answer the customer's question directly using the retrieved information. Quote relevant sentences or sections if helpful, but primarily summarize the answer in your own words.
6.  If the retrieved information is unclear or doesn't fully answer the question, state what you found and what remains unanswered. Do NOT invent policy details.
7.  If the `get_policy_info` tool returns no relevant policy, inform the customer that you couldn't find specific information on that topic in the policy documents and suggest they contact human support for clarification.

Interaction Snippet:

• User: "Hi, can I get a refund if I've opened the product box?"
• Agent's internal thought process (guided by the prompt):
  • This is a policy question about returns and product condition.
  • Keywords: "refund", "opened box", "return condition".
  • Query for get_policy_info: "return policy opened product refund"
• Agent calls get_policy_info("return policy opened product refund")
• Tool returns: "Policy 4.2.b: Products may be returned within 30 days if in original, unopened packaging. Products that have been opened may be eligible for exchange or store credit at manager's discretion if all original components are present and the product is undamaged. Opened products are generally not eligible for a full cash refund unless defective."
• Agent's response (guided by prompt on how to use info): "According to our policy, products that have been opened are generally not eligible for a full cash refund unless they are defective. However, an opened product might be eligible for an exchange or store credit if it's undamaged and all original components are present, though this is often at the manager's discretion. Would you like information on how to check for eligibility for an exchange or store credit?"

This example shows how detailed prompting guides the agent not just to use a tool, but to do so intelligently, from query formulation to the nuanced presentation of retrieved information.

Adapting Prompts to Knowledge Store Types

The specific phrasing of your prompts for query formulation will naturally vary depending on the type of long-term storage:

• Vector Databases (Semantic Search): Prompts might encourage natural language queries or questions that capture the user's intent, as these databases excel at understanding semantic similarity.

  To find relevant documents, formulate a full question that captures the essence of what the user is asking for.
  

• SQL Databases: Prompts must guide the agent to construct syntactically correct SQL queries, potentially by providing table schemas or examples.

  Generate a SQL query for the `analytics_db` to retrieve the requested data. The 'sales' table has columns: 'product_id', 'sale_date', 'amount', 'region'.
  

• Graph Knowledge Bases (e.g., SPARQL): Prompts would need to guide the formation of graph query language statements.
• Document Management Systems with Keyword Search: Prompts would focus on extracting the most salient keywords.

While the tool-specific details matter, the underlying principle is consistent: the prompt translates the agent's understanding of the task into an effective interaction with the knowledge store.

Retrieving information from long-term memory is a dynamic process. You will likely iterate on your prompts, observing how the agent formulates queries, how it interprets results, and how well it integrates new knowledge. This iterative refinement is key to building highly capable and reliable AI agents. As you move to the next section, "Structuring Prompts for Accessing Knowledge Stores," you'll learn more about defining the tools and their descriptions within the prompt so the agent can understand how to call them in the first place.