Practice: Developing and Testing Prompts

Prompt engineering is largely an empirical process. While theoretical understanding is valuable, practical application involves writing a prompt, observing the Large Language Model's (LLM) output, identifying shortcomings, and iteratively refining the prompt until desired results are consistently produced. The hands-on refinement process using Python is detailed, guiding users through developing and testing prompts.

We'll work through a common task: extracting structured information from unstructured text. Imagine you have product descriptions, and you want to pull out the specific features mentioned.

Scenario: Extracting Product Features

Let's say we have the following product description for a fictional camera:

product_description = """
Introducing the PhotoMax Pro, a revolutionary new camera for enthusiasts.
Important specs include a 30MP full-frame sensor, advanced autofocus with 500 points,
and 8K video capability. It boasts a weather-sealed magnesium alloy body,
dual SD card slots, and built-in Wi-Fi and Bluetooth for easy sharing.
The high-resolution electronic viewfinder provides a crystal-clear preview.
"""

Our goal is to extract a clean list of features from this text.

Attempt 1: Basic Zero-Shot Prompt

We'll start with the simplest approach: asking the LLM directly without providing examples or specific formatting instructions. We can use the OpenAI library (assuming you have it installed and your API key configured as shown in Chapter 2) or a similar client library.

# Using the OpenAI library (example)
# Ensure 'openai' is installed and API key is set in environment variables
import os
from openai import OpenAI

# Make sure OPENAI_API_KEY environment variable is set
# client = OpenAI() # Automatically reads from environment variable
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def extract_features_attempt_1(text):
    prompt = f"Extract the main features from the following product description:\n\n{text}"
    try:
        response = client.chat.completions.create(
            model="gpt-3.5-turbo", # Or another suitable model
            messages=[
                {"role": "user", "content": prompt}
            ],
            temperature=0.2 # Lower temperature for more deterministic output
        )
        return response.choices[0].message.content
    except Exception as e:
        print(f"An API error occurred: {e}")
        return None

# Run the first attempt
features_v1 = extract_features_attempt_1(product_description)
print("--- Attempt 1 Output ---")
print(features_v1)

Potential Output (V1):

--- Attempt 1 Output ---
Here are the main features of the PhotoMax Pro:
- 30MP full-frame sensor
- Advanced autofocus with 500 points
- 8K video capability
- Weather-sealed magnesium alloy body
- Dual SD card slots
- Built-in Wi-Fi and Bluetooth
- High-resolution electronic viewfinder

Analysis (V1):

This isn't bad! The LLM understood the request and extracted most of the important features. However, the output includes introductory text which might not be ideal if we want to directly parse this list in our application. The formatting is decent (bullet points), but we didn't explicitly ask for it, so we can't rely on it being consistent across different inputs or models.

Attempt 2: Adding Structure and Output Format Instructions

Let's refine the prompt to be more specific about the desired output format. We'll explicitly ask for a comma-separated list.

# Using LangChain's PromptTemplate for better structure
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

# Assume OPENAI_API_KEY is set
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0.2)

prompt_template_v2 = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert assistant that extracts product features from text.",
        ),
        (
            "human",
            "Extract the main features from the product description below. "
            "List only the features, separated by commas. Do not include introductory text.\n\n"
            "Description:\n{product_desc}",
        ),
    ]
)

chain_v2 = prompt_template_v2 | llm

def extract_features_attempt_2(text):
    try:
        response = chain_v2.invoke({"product_desc": text})
        return response.content
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

# Run the second attempt
features_v2 = extract_features_attempt_2(product_description)
print("\n--- Attempt 2 Output ---")
print(features_v2)

Potential Output (V2):

--- Attempt 2 Output ---
30MP full-frame sensor, advanced autofocus with 500 points, 8K video capability, weather-sealed magnesium alloy body, dual SD card slots, built-in Wi-Fi, built-in Bluetooth, high-resolution electronic viewfinder

Analysis (V2):

This is much better for programmatic use. We got rid of the introductory phrase and achieved the desired comma-separated format. Using ChatPromptTemplate (as shown in Chapter 4) also makes the prompt structure clearer, separating system instructions from the user's request. The separation of "Wi-Fi and Bluetooth" into distinct features is also a good improvement.

Attempt 3: Few-Shot Prompting for Consistency

While Attempt 2 worked well for this specific description, LLMs can sometimes struggle with nuances or variations in input style. Adding a few examples (few-shot prompting) can significantly improve robustness and guide the model towards the exact desired output style, especially for more complex extractions.

Let's add an example to our LangChain prompt template.

# Using LangChain with Few-Shot examples
prompt_template_v3 = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert assistant that extracts product features from text. "
            "List only the features, separated by commas. Do not include introductory text.",
        ),
        # Few-shot example 1
        (
            "human",
            "Description:\nThe new SoundWave headphones feature noise cancellation and a 20-hour battery life."
        ),
        (
            "ai",
            "noise cancellation, 20-hour battery life"
        ),
         # Few-shot example 2 (Optional, adding more examples can help)
        (
            "human",
            "Description:\nThis laptop comes with a 1TB SSD, 16GB RAM, and a backlit keyboard."
         ),
        (
            "ai",
            "1TB SSD, 16GB RAM, backlit keyboard"
        ),
        # Actual request
        (
            "human",
            "Description:\n{product_desc}",
        ),
    ]
)

chain_v3 = prompt_template_v3 | llm # llm defined in Attempt 2

def extract_features_attempt_3(text):
    try:
        response = chain_v3.invoke({"product_desc": text})
        return response.content
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

# Run the third attempt
features_v3 = extract_features_attempt_3(product_description)
print("\n--- Attempt 3 Output ---")
print(features_v3)

Potential Output (V3):

--- Attempt 3 Output ---
30MP full-frame sensor, advanced autofocus with 500 points, 8K video capability, weather-sealed magnesium alloy body, dual SD card slots, built-in Wi-Fi, built-in Bluetooth, high-resolution electronic viewfinder

Analysis (V3):

In this case, the output might be identical to Attempt 2 because the zero-shot prompt with clear instructions already performed well on this input. However, the few-shot examples provide stronger guidance. If the input description were more complex, ambiguous, or written in a different style, these examples would significantly increase the likelihood of getting the correctly formatted, relevant features. They demonstrate the expected input-output pattern directly to the model.

The Iterative Refinement Process

This example demonstrates a common workflow:

1. Start Simple: Begin with a clear, direct prompt.
2. Execute & Observe: Run the prompt with representative input data using your Python code.
3. Analyze Output: Compare the LLM's response against your requirements. Identify errors, omissions, or formatting issues.
4. Refine Prompt: Modify the prompt based on your analysis. This might involve:
   • Adding more specific instructions.
   • Clarifying the desired output format.
   • Providing few-shot examples.
   • Rewording instructions for better clarity.
   • Adjusting model parameters like temperature.
5. Repeat: Go back to step 2 with the refined prompt and test again, potentially using different input data.

The iterative cycle of prompt development: define, prompt, execute, observe, analyze, and refine until the goal is met.

Testing with Diverse Inputs

A prompt that works perfectly for one input might fail on another. It's essential to test your refined prompt (like prompt_template_v3) against a variety of inputs:

• Different Product Types: Try descriptions of laptops, phones, software, etc.
• Varying Lengths: Use both short and long descriptions.
• Different Styles: Include marketing-heavy text vs. technical specification lists.
• Edge Cases: Test descriptions with no clear features or ambiguous wording.

test_descriptions = [
    "Our new coffee maker brews in under a minute and has a programmable timer.", # Simple case
    "The TrailSeeker backpack: 50L capacity, integrated rain cover, adjustable torso length, hydration compatible.", # List format
    "Experience blazing speed with the Quantum Processor X1. Features include 128GB storage, a 6.8-inch AMOLED display, and 5G connectivity.", # Different product
    "Just a basic t-shirt. 100% cotton.", # Few features
    "Innovative solution using next-generation architecture for improved user workflow."
]

print("\n--- Testing Prompt V3 with Diverse Inputs ---")
for i, desc in enumerate(test_descriptions):
    print(f"\nInput {i+1}: {desc}")
    features = extract_features_attempt_3(desc) # Using our best prompt so far
    print(f"Output {i+1}: {features}")

Running these tests helps you gauge the robustness of your prompt and identify areas for further refinement if needed. This forms the basis for more structured evaluation, which we cover in the next chapter.

Dynamic Prompt Generation in Python

As mentioned earlier, Python makes it easy to create prompts dynamically. Imagine you only want features related to connectivity:

def extract_specific_features(text, feature_category):
    # Note: This simple approach might need more sophisticated prompting for reliability
    # It might be better to extract all features first, then filter.
    # But it demonstrates dynamic insertion.
    prompt = ChatPromptTemplate.from_messages(
        [
            (
                "system",
                "You are an expert assistant that extracts specific product features.",
            ),
            (
                "human",
                f"Extract only the features related to '{feature_category}' from the description below. "
                "List only the relevant features, separated by commas.\n\n"
                "Description:\n{product_desc}",
            ),
        ]
    )
    chain = prompt | llm # llm defined earlier
    try:
        response = chain.invoke({"product_desc": text, "feature_category": feature_category})
        return response.content
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

connectivity_features = extract_specific_features(product_description, "connectivity")
print("\n--- Dynamic Prompt: Connectivity Features ---")
print(connectivity_features)

sensor_features = extract_specific_features(product_description, "sensor or image quality")
print("\n--- Dynamic Prompt: Sensor Features ---")
print(sensor_features)

Potential Output:

--- Dynamic Prompt: Connectivity Features ---
built-in Wi-Fi, built-in Bluetooth

--- Dynamic Prompt: Sensor Features ---
30MP full-frame sensor, advanced autofocus with 500 points

This demonstrates how Python variables can be integrated into your prompts using f-strings or templating engines like LangChain's, allowing for flexible and context-aware interactions with the LLM.

This practical exercise highlights that prompt engineering is an active process. By combining clear instructions, illustrative examples, and iterative testing within your Python environment, you can significantly improve the quality and reliability of your LLM application's outputs. Remember to test thoroughly and be prepared to refine your prompts as you encounter new data or requirements.