When building LLM applications that are used frequently, you will face two practical challenges: latency and operational cost. Each call to a language model or an embedding service takes time to process and incurs a fee, which is often calculated based on the number of tokens. The total cost $C$ for a single API call can be represented as:

$C = (P_{prompt} \times N_{prompt}) + (P_{completion} \times N_{completion})$

where $P$ is the price per token and $N$ is the number of tokens for the prompt and completion. For applications that handle repetitive queries, these costs can accumulate quickly.

This chapter introduces caching as a primary technique to address these issues. We will start by identifying common performance bottlenecks in an LLM-driven system. You will then learn to implement two specific caching strategies using the cache module:

• LLM Response Caching: Storing and reusing complete model generations for identical prompts.
• Embedding Caching: Avoiding redundant API calls to generate vector embeddings for the same pieces of text.

Finally, we will cover the practical matter of cache invalidation to ensure your application's data remains current. By working through these sections, you will learn how to make your applications faster and more cost-effective.