Practice: Cost Modeling for a Sample RAG Application

Building a cost model for a sample Retrieval-Augmented Generation (RAG) application helps identify significant cost contributors and understand how different choices impact overall expenditure.

Scenario: "IntelliDocs" Q&A System

Imagine "IntelliDocs," an internal Q&A system designed to help employees find answers from a large repository of technical documentation, API guides, and engineering wikis.

System Specifications:

• Knowledge Base Volume:
  • Initial documents: 100,000
  • Average document length: 1,500 words (approximately 2,000 tokens)
  • Chunking strategy: Overlapping chunks of 512 tokens each. This results in roughly $100,000 \text{ docs} \times (2000 \text{ tokens/doc} / 512 \text{ tokens/chunk}) \approx 390,625$ chunks. Let's round to 400,000 chunks for simplicity.
  • Monthly updates: 5% of documents are new or revised, requiring re-embedding of affected chunks.
• User Activity:
  • Queries per month: 50,000
  • Average query length: 30 tokens
  • Average retrieved chunks per query: 3
  • Average generated response length: 200 tokens
• Model Choices (for estimation purposes):
  • Embedding Model:
    • Option 1 (Self-hosted Open Source, e.g., sentence-transformers/all-mpnet-base-v2): Primarily compute cost. For this model, let's assume a simplified operational cost for embedding that equates to $0.00005 per 1k tokens (covering compute, maintenance).
    • Option 2 (Proprietary API, e.g., OpenAI text-embedding-ada-002): $0.0001 per 1k tokens.
  • Generator LLM (API-based):
    • Option A (High-End LLM, e.g., GPT-4 class): $0.03 per 1k prompt tokens,$0.06 per 1k completion tokens.
    • Option B (Mid-Tier LLM, e.g., GPT-3.5-Turbo class): $0.001 per 1k prompt tokens,$0.002 per 1k completion tokens.
• Infrastructure:
  • Vector Database: Managed service with storage and query costs.
  • Application Logic: Serverless functions (e.g., AWS Lambda).
  • Logging/Monitoring: Standard cloud services.

Step 1: Identify Primary Cost Components

For IntelliDocs, the main cost drivers will be:

1. Initial Data Ingestion: Embedding the entire knowledge base.
2. Ongoing Data Updates: Embedding new or modified documents.
3. Vector Database: Storage of embeddings and query operations.
4. Query Processing:
   • Embedding user queries.
   • LLM API calls for generation.
5. Compute/Orchestration: Running the application logic (e.g., serverless functions, API gateways).
6. Data Storage (Raw Documents & Logs): Storing original documents and system logs.
7. Monitoring: Costs associated with monitoring tools and services.

Step 2: Estimate Costs for Each Component

Let's use Embedding Model Option 2 (Proprietary API) for initial calculations to simplify, then compare. We'll analyze Generator LLM Options A and B.

2.1 Initial Data Ingestion (Embedding Costs)

• Total chunks: 400,000
• Tokens per chunk: 512
• Total tokens for initial ingestion: $400,000 \text{ chunks} \times 512 \text{ tokens/chunk} = 204,800,000 \text{ tokens}$
• Cost with Embedding Model Option 2 (API): (204,800,000 / 1000) \times \0.0001 = $20.48$
  • This is a one-time cost.

2.2 Ongoing Data Updates (Monthly Embedding Costs)

• Documents to update: $5\% \times 100,000 = 5,000 \text{ documents}$
• Assuming each updated document requires re-embedding its chunks (average 4 chunks/document): $5,000 \text{ docs} \times 4 \text{ chunks/doc} = 20,000 \text{ chunks}$
• Tokens for updates: $20,000 \text{ chunks} \times 512 \text{ tokens/chunk} = 10,240,000 \text{ tokens}$
• Monthly update cost (Embedding Model Option 2): (10,240,000 / 1000) \times \0.0001 = $1.02$

2.3 Vector Database Costs (Monthly)

Vector database pricing is highly variable. Let's assume a simplified model for a managed service:

• Storage: 400,000 vectors, 768 dimensions (e.g., all-mpnet-base-v2 or ada-002), float32 (4 bytes/dimension).
  • Size per vector: $768 \times 4 \text{ bytes} = 3072 \text{ bytes}$
  • Total storage: $400,000 \times 3072 \text{ bytes} \approx 1.23 \text{ GB}$
  • Let's estimate storage and basic operational cost for a managed vector DB at $50 per month for this scale. This is a rough estimate; actual costs can vary significantly based on provider, features, and performance tiers. Some providers might charge per million vectors stored or based on instance hours.

2.4 Query Processing Costs (Monthly)

2.4.1 Query Embedding Costs

• Queries per month: 50,000
• Average query length: 30 tokens
• Total query tokens: $50,000 \times 30 = 1,500,000 \text{ tokens}$
• Monthly query embedding cost (Embedding Model Option 2): (1,500,000 / 1000) \times \0.0001 = $0.15$

2.4.2 LLM Generation Costs

This is often the most significant recurring cost.

• Number of queries: 50,000
• Context tokens per query:
  • Query: 30 tokens
  • Retrieved Chunks: $3 \text{ chunks} \times 512 \text{ tokens/chunk} = 1536 \text{ tokens}$
  • Total prompt tokens per query: $30 + 1536 = 1566 \text{ tokens}$
• Completion tokens per query: 200 tokens

Using Generator LLM Option A (High-End):

• Prompt cost per query: (1566 / 1000) \times \0.03 = $0.04698$
• Completion cost per query: (200 / 1000) \times \0.06 = $0.012$
• Total cost per query (Option A): \0.04698 + $0.012 = $0.05898$
• Monthly LLM cost (Option A): 50,000 \times \0.05898 = \textbf{$2949.00}$

Using Generator LLM Option B (Mid-Tier):

• Prompt cost per query: (1566 / 1000) \times \0.001 = $0.001566$
• Completion cost per query: (200 / 1000) \times \0.002 = $0.0004$
• Total cost per query (Option B): \0.001566 + $0.0004 = $0.001966$
• Monthly LLM cost (Option B): 50,000 \times \0.001966 = \textbf{$98.30}$

2.5 Compute/Orchestration Costs (Monthly)

For serverless functions processing 50,000 requests, with each request involving embedding, vector search, and LLM API calls, the compute duration might be a few seconds.

• Let's estimate an average of 2 seconds per request, using 512MB RAM functions.
• Total compute-seconds: $50,000 \times 2s = 100,000 \text{ GB-seconds}$ (assuming 1GB RAM equivalence for pricing, adjust based on actual provider tiers).
• A typical serverless function cost might be around $0.00001667 per GB-second.
• Monthly compute cost: 100,000 \times \0.00001667 \approx $1.67$.
• API Gateway costs: For 50,000 requests, this might be around $2-5 per month.
• Total estimated orchestration: $10 per month. This is highly dependent on the specific architecture and cloud provider.

2.6 Data Storage (Raw Documents & Logs) (Monthly)

• Raw documents: 100,000 documents, average 1500 words. If each word is ~5 chars, and docs are text: $100,000 \times 1500 \times 5 \text{ bytes} \approx 750 \text{ MB}$.
• Logs: Dependent on verbosity. Let's estimate 10GB of logs per month.
• Standard cloud storage (e.g., S3/GCS): \approx \0.023 \text{ per GB/month}$.
• Monthly storage cost: (0.75 \text{ GB} + 10 \text{ GB}) \times \0.023 \approx $0.25$. Let's round to **$1 per month**.

2.7 Monitoring Costs (Monthly)

• Basic cloud monitoring services for metrics, dashboards, and alerts might range from $10 to$50 per month for this scale, depending on the granularity and retention. Let's use $20 per month.

Step 3: Summarize Monthly Costs

Let's create a summary table. We'll use Embedding Model Option 2 (API-based) for embedding costs.

Cost Component	Monthly Cost (LLM Option A: High-End)	Monthly Cost (LLM Option B: Mid-Tier)	Notes
Embedding Updates	$1.02	$1.02	API-based embedding model
Vector Database	$50.00	$50.00	Estimate for managed service
Query Embedding	$0.15	$0.15	API-based embedding model
LLM Generation	$2949.00	$98.30	Significant difference based on model choice
Compute/Orchestration	$10.00	$10.00	Serverless functions, API Gateway
Data Storage (Raw/Logs)	$1.00	$1.00
Monitoring	$20.00	$20.00
Total Estimated Monthly Cost	$3031.17	$180.47

The initial one-time ingestion cost was $20.48.

Step 4: Visualizing Cost Impact

A simple visualization can highlight the most impactful cost components. The LLM generation cost clearly dominates, especially with the high-end model.

Estimated total monthly operational costs for the IntelliDocs RAG system, comparing a high-end generator LLM (Option A) versus a mid-tier generator LLM (Option B).

Step 5: Analyzing the Model and Identifying Optimization Levers

This cost model, though simplified, reveals several important points:

1. LLM Generation is Dominant: The choice of generator LLM and the number of tokens processed per query are by far the largest cost drivers.
   • Optimization: Implementing strategies from "Techniques for Minimizing LLM Token Usage" (e.g., prompt compression, context window optimization, asking the LLM for more concise answers) is critical. Switching to a more cost-effective LLM (Option B) yields a dramatic reduction (over 90% in this example). Fine-tuning smaller, open-source models for specific tasks could offer even greater savings if feasible.
2. Embedding Costs: While not as high as LLM generation in this scenario, embedding costs can become substantial with very large datasets or frequent updates, especially if using API-based embedding models.
   • Optimization: Consider self-hosting open-source embedding models (like our Option 1 costing $0.00005/1k tokens). For ongoing updates, this would be$0.51/month instead of $1.02/month. For query embeddings, it'd be negligible. The initial ingestion cost would be$10.24 instead of $20.48. While these savings are small here, they scale with volume. The trade-off is operational overhead.
3. Vector Database: Costs can vary widely. For very large systems, optimizing indexing strategies, choosing appropriate instance types, or considering sharding (as discussed in "Vector Database Optimization") can lead to savings.
4. Compute/Orchestration: Serverless is often cost-effective for variable loads. However, for very high, sustained throughput, provisioned resources might become more economical. Batching requests can also reduce per-request overhead.
5. Caching: Implementing caching for LLM responses (for identical queries with identical context) or frequently accessed retrieved documents could reduce LLM calls and vector DB lookups, directly impacting costs. If 10% of queries could be served from a cache, that’s a direct 10% saving on the LLM generation cost for those queries.

Building Your Own Cost Model

This exercise provides a template. To model costs for your RAG application:

1. Define Your Scenario: Detail your data volume, update frequency, expected query load, and performance needs.
2. List Components: Identify all services and operations that incur costs (embedding, vector DB, LLM APIs, compute, storage, monitoring, etc.).
3. Gather Pricing: Obtain current pricing for your chosen cloud services and model APIs. Be aware that prices can change.
4. Estimate Usage: Quantify your usage for each component (e.g., number of tokens, API calls, storage GB, compute hours).
5. Calculate Costs: Use a spreadsheet or a simple script to calculate costs for each component and sum them up.
   • Input variables: queries/month, avg_prompt_tokens, avg_completion_tokens, embedding_cost_per_token, llm_prompt_cost_per_token, llm_completion_cost_per_token, etc.
   • Formulas: total_llm_cost = queries_per_month * ((avg_prompt_tokens * llm_prompt_cost_per_token) + (avg_completion_tokens * llm_completion_cost_per_token))
6. Analyze and Iterate: Identify the largest cost contributors. Explore how different architectural choices, model selections, or optimization techniques (like those discussed in this course) would affect the total cost. For example, what if you reduce average prompt tokens by 20% through better context selection?

Conclusion of Practice

Cost modeling is an iterative process. Your initial model will be an estimate, but as you gain more understanding of your system's usage patterns and explore different configurations, you can refine it. Regularly revisiting your cost model, especially when considering system changes or scaling, is essential for maintaining a cost-efficient RAG system in production. This practice equips you with a structured approach to anticipate, analyze, and manage these operational expenses.