Prompt Strategies for Short-Term Memory and Context Windows

An AI agent's immediate recall, its ability to remember what just happened or what its current specific instruction is, relies heavily on the Large Language Model's (LLM) context window. This window acts as the agent's short-term or working memory, holding the information the LLM can directly access for generating its next response or action. Effective prompt engineering is essential for managing this finite resource, ensuring the agent maintains coherence and focuses on the relevant details during multi-step operations.

The Context Window: An Agent's Working Memory

Think of the LLM's context window like the RAM in a computer: it's fast, directly accessible memory, but it has a limited capacity. Every piece of information provided to an LLM in a single interaction. This includes system instructions defining the agent's role, the user's current query, a transcript of recent conversation turns, descriptions of available tools, and any intermediate thoughts or "scratchpad" notes. Must fit within this window.

The size of the context window is measured in tokens, which can roughly be thought of as words or parts of words. Models come with different context window sizes, for example, 4,096 tokens, 8,192 tokens, 32,768 tokens, or even larger. While larger windows offer more space, they are always finite. If the information stream exceeds this limit, older information is typically pushed out, leading to the agent "forgetting" earlier parts of the interaction. This can disrupt task continuity and lead to errors or irrelevant responses.

For instance, if an agent's context window is 4096 tokens:

System instructions and role definition might consume 200 tokens.
Descriptions of available tools might take up 500 tokens.
This leaves $4096 - 200 - 500 = 3396$ tokens for the actual conversational history, user queries, and any dynamic working notes for the current turn.

Managing this budget effectively through prompt design is a core skill in building capable agents.

Core Prompt Strategies for Managing the Context Window

To make the most of an agent's short-term memory, you can employ several prompt engineering strategies. These techniques help ensure that the most relevant information stays within the LLM's attention span.

1. Strategic Information Placement

LLMs don't always weigh all parts of the prompt equally. Often, information at the very beginning (primacy effect) or the very end (recency effect) of the prompt has a stronger influence on the output.

Primacy: Place foundational information, such as the agent's core persona, overarching goals, or critical constraints, at the beginning of your prompt structure. This is often part of the "system prompt" if the underlying LLM API supports it, or the initial, fixed part of a user prompt.
```
System: You are a meticulous research assistant. Your goal is to gather information about renewable energy sources. Always cite your sources. Do not provide opinions, only factual data.
```
Recency: The most immediate task, the user's latest question, or specific instructions for the current step should be positioned towards the end of the input. This ensures it's the last thing the model processes before generating a response.
```
User: ... [previous conversation history summarized] ...
User: Now, based on the findings so far, identify the top three challenges in solar panel adoption in urban environments.
```

2. Maintaining Conversational Flow with History Management

For an agent to hold a coherent conversation or execute a sequence of related tasks, it needs to remember what was said or done previously.

Rolling History Window: Include a transcript of the last $N$ turns of the conversation in the prompt. This provides immediate historical context. The value of $N$ depends on the token budget and the nature of the task.

Summarization Instructions: As the conversation grows, the full history will exceed the context window. You can instruct the agent (or use a separate LLM call) to summarize earlier parts of the conversation. This compressed summary is then included in subsequent prompts.

<system_prompt>
You are a helpful assistant.
</system_prompt>

<user_prompt>
<conversation_history_summary>
The user is planning a 7-day trip to Paris for 2 adults in spring. Budget is moderate. Interests include museums and historical sites.
</conversation_history_summary>

<recent_exchanges>
User: What about evening entertainment options?
Agent: For evening entertainment, Paris offers classical concerts, theatre shows, and Seine river cruises. Do any of these sound appealing?
User: The river cruise sounds nice. Can you find options that include dinner?
</recent_exchanges>

Current question: Find dinner cruise options on the Seine for two adults.
</user_prompt>

In this example, conversation_history_summary is a condensed version of earlier turns, while recent_exchanges keeps the last few interactions verbatim.

3. The "Scratchpad" or "Working Notes" Technique

You can instruct an agent to maintain a "scratchpad" or a "working_notes" section directly within its prompt. This section serves as a mutable space where the agent can jot down intermediate thoughts, calculations, observations, or refined plans.

The agent is prompted to read from and update this scratchpad as part of its reasoning process. After the agent generates its response (which includes the updated scratchpad), your controlling application code extracts this scratchpad and includes it in the prompt for the next turn.

Benefits:

Makes the agent's reasoning process more transparent.
Helps the agent keep track of complex state information or partial results.
Reduces the chance of forgetting intermediate steps.

Example Prompt Structure:

System: You are an AI assistant tasked with solving a multi-step problem. Use the <scratchpad> to think step-by-step and record important information.

User:
<problem_description>
[Detailed problem statement here]
</problem_description>

<scratchpad>
[Previous scratchpad content, or "Think step-by-step here..." for the first turn]
</scratchpad>

Based on the problem and your scratchpad, what is the next step or your final answer? Update your scratchpad.

The agent's output would include its reasoning and the updated content for <scratchpad>, which is then fed back into the next prompt.

4. Structured Data for Clarity

Using clear delimiters, such as XML-like tags (e.g., <history>, <user_query>, <available_tools>) or Markdown headings, helps the LLM differentiate between various types of information within a complex prompt. This can improve the model's ability to parse the prompt accurately and focus on the sections most relevant to its current sub-task.

### Instructions
You are a helpful assistant.

### Available Tools
- search_web(query: string)
- get_weather(city: string)

### Conversation History
User: What's the weather like in London?
Agent: The weather in London is currently 15°C and cloudy.
User: Thanks! Now, can you search for popular tourist attractions there?

### Current Task
Based on the conversation, use the appropriate tool to find popular tourist attractions in London.

This structured approach aids the LLM in understanding the different components of its input and their roles.

5. Explicit Context Injection

While the above strategies manage information already introduced into the interaction, sometimes the agent needs specific pieces of information that it hasn't encountered yet or that were part of a much earlier, now summarized, interaction. Your application logic can proactively inject relevant context snippets into the prompt. For example, if the agent mentions a specific entity, the system can retrieve a brief definition or important facts about that entity from a knowledge base and add it to the prompt for the current turn, enriching the agent's short-term memory.

Visualizing Context Window Components

The following diagram illustrates how various pieces of information constitute the prompt, which in turn populates the LLM's context window for a given processing turn.

The diagram shows that the prompt, containing system instructions, user requests, history, tool data, and working notes, fills the LLM's context window. This window then informs the LLM's processing unit to generate an output.

Addressing Context Overflow Proactively

The primary goal of these prompt strategies is to use the limited context window efficiently and prevent critical information from being lost. When the volume of information (system prompts, history, tool descriptions, current query) exceeds the LLM's token limit, the model will inevitably "forget" some of it, usually the oldest information in a simple truncation scenario. This loss can cause the agent to lose track of its goals, repeat previous actions, or fail to use relevant past information.

By actively managing what goes into the prompt each turn, summarizing, selecting, and structuring, you reduce the likelihood of such context overflow. These techniques are your first line of defense for maintaining coherent, stateful agent behavior over several turns of interaction.

While these methods are powerful for managing short-term memory, they are not a complete solution for long-term knowledge retention or recalling information from datasets. For those requirements, strategies involving external knowledge bases and retrieval mechanisms, discussed later in this chapter, become essential. However, a well-managed context window is the foundation upon which all other memory techniques build.

Build LLM apps faster with Kerb

Cleaner syntax. Built-in debugging. Production-ready from day one.

Built for the AI systems behind ApX Machine Learning

Was this section helpful?

References

Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017 arXiv preprint arXiv:1706.03762 DOI: 10.48550/arXiv.1706.03762 - Foundational paper introducing the Transformer architecture, which forms the basis for current large language models and their context window mechanism.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022 arXiv (arXiv) DOI: 10.48550/arXiv.2201.11903 - Introduces Chain-of-Thought prompting, a method directly relevant to the scratchpad technique for enhancing model reasoning and intermediate thought tracking within the context window.
Lost in the Middle: How Language Models Use Long Contexts, Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 Transactions of the Association for Computational Linguistics (TACL) DOI: 10.48550/arXiv.2307.03172 - Research exploring how large language models process information across long contexts, detailing phenomena such as primacy and recency effects that influence strategic information placement in prompts.
ReAct: Synergizing Reasoning and Acting in Language Models, Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2023 arXiv preprint arXiv:2210.03629 DOI: 10.48550/arXiv.2210.03629 - Presents ReAct, an agentic framework combining reasoning and acting through a structured scratchpad approach, demonstrating effective context management for multi-step agent tasks.
Prompt Engineering Guide, Anthropic, 2024 (Anthropic) - Official guide offering practical strategies and best practices for prompt engineering with large language models, including advice on managing context, structuring prompts, and token awareness.