\"\"\" clean_content = extract_text_from_html(html_content) print(clean_content) # Main Title # This is important content.Similarly, if you are working with Markdown, you might want to strip the formatting to get to the underlying text. The strip_markdown function can be used for this purpose, converting **bold** text into just bold.Removing Irrelevant and Sensitive ContentFor many RAG applications, elements like URLs, email addresses, and phone numbers are noise that can distort the semantic meaning of a chunk. Removing them helps the embedding model focus on the primary content. More importantly, removing or redacting Personally Identifiable Information (PII) is a standard practice for privacy and data protection.You can remove these elements individually or as part of a larger cleaning pipeline. For sensitive applications, instead of just removing PII, you might choose to redact it, replacing the sensitive data with a generic placeholder.from kerb.safety import redact_pii sensitive_text = \"Please contact support at [email protected] or call 555-123-4567 for help.\" redacted_text = redact_pii(sensitive_text) print(f\"Original: {sensitive_text}\") print(f\"Redacted: {redacted_text}\") # Original: Please contact support at [email protected] or call 555-123-4567 for help. # Redacted: Please contact support at [REDACTED_EMAIL] or call [REDACTED_PHONE] for help.This approach preserves the context that a piece of information was present while protecting the sensitive data itself.Building a Preprocessing PipelineWhile you can apply these functions one by one, it's more efficient to define a reusable preprocessing pipeline. The normalize_text function, combined with a NormalizationConfig, allows you to orchestrate multiple cleaning and normalization steps in a declarative way. You can define different configurations for different types of content or stages of your RAG pipeline.Let's build a standard pipeline for processing web-scraped articles. The goal is to lowercase the text, remove URLs and emails, and normalize all whitespace and quotes.from kerb.preprocessing import normalize_text, NormalizationConfig, NormalizationLevel # Define a configuration for our pipeline web_article_config = NormalizationConfig( level=NormalizationLevel.STANDARD, # Applies standard unicode, quote, and whitespace normalization lowercase=True, remove_urls=True, remove_emails=True, remove_extra_spaces=True ) messy_article_snippet = \"\"\" Check out our new \"Report\" at https://example.com/report! Contact [email protected] for info. It's an in-depth analysis of AI’s impact. \"\"\" # Apply the entire pipeline in one call processed_text = normalize_text(messy_article_snippet, config=web_article_config) print(\"### Before Preprocessing ###\") print(messy_article_snippet) print(\"\\n### After Preprocessing ###\") print(processed_text)This \"before and after\" comparison shows the result of our pipeline:Before PreprocessingCheck out our new \"Report\" at https://example.com/report! Contact [email protected] for info. It's an in-depth analysis of AI’s impact.After Preprocessingcheck out our new \"report\" at ! contact for info. it's an in-depth analysis of ai's impact.The output is now a clean, standardized block of text. This cleaned version is far more suitable for an embedding model, as the noise has been stripped away, leaving only the core textual content. The level of preprocessing is a balance. Overly aggressive cleaning might remove important context, so it's always good practice to test the impact of your pipeline on retrieval quality.With our documents loaded, chunked, and now thoroughly cleaned, we have a high-quality dataset of text fragments. Each fragment is a semantically coherent and normalized unit of information, ready for the next step: converting text into numerical representations through embeddings.","isAccessibleForFree":false,"hasPart":{"@type":"WebPageElement","isAccessibleForFree":false,"cssSelector":".login-required-content"}}

Text Preprocessing for Better Retrieval

Cleaning and standardizing text is a critical phase in preparing data for effective retrieval. The quality of your retrieval system is directly tied to the quality of the data it searches. Raw text, especially from sources like web pages or scanned documents, is often filled with noise that can degrade the performance of embedding models. This noise includes formatting artifacts, inconsistent whitespace, irrelevant metadata, and sensitive information.

Preprocessing aims to create a clean, consistent representation of the text that accurately captures its semantic meaning. By removing distracting elements, you help the embedding model focus on the content that matters, leading to more precise and relevant retrieval results. This section will guide you through the common techniques for normalizing and cleaning your text data.

Normalizing Text for Consistency

Text normalization is the process of transforming text into a single, canonical form. This is important because embedding models are sensitive to variations in input that might not change the meaning for a human reader. For instance, "Hello World" and "hello world" should ideally be represented similarly if case is not a significant factor for your application.

A common starting point is to handle whitespace. Text from different sources can have erratic spacing, multiple newlines, or tabs, which can introduce noise into your embeddings. You can standardize this by collapsing multiple whitespace characters into a single space and normalizing line breaks.

from kerb.preprocessing import normalize_whitespace

raw_text = "This text    has extra spaces\n\n\nand multiple newlines."
normalized = normalize_whitespace(raw_text)

print(f"Original: {repr(raw_text)}")
print(f"Normalized: {repr(normalized)}")
# Original: 'This text    has extra spaces\n\n\nand multiple newlines.'
# Normalized: 'This text has extra spaces and multiple newlines.'

Another subtle but important task is Unicode normalization. Characters like "é" can be represented in multiple ways in Unicode. Normalizing them ensures that the same character is always represented by the same byte sequence, which is significant for consistent tokenization.

Cleaning Source-Specific Artifacts

Different document formats introduce unique types of noise. Web pages are filled with HTML tags, scripts, and CSS, while text extracted from PDFs might contain hyphenation artifacts from line breaks and irregular spacing. Removing this structural noise is a primary step.

For HTML content, you can extract the clean, human-readable text while discarding all the markup.

from kerb.preprocessing import extract_text_from_html

html_content = """
<html>
    <body>
        <h1>Main Title</h1>
        <p>This is <strong>important</strong> content.</p>
        <script>alert('noise');</script>
    </body>
</html>
"""

clean_content = extract_text_from_html(html_content)
print(clean_content)
# Main Title
# This is important content.

Similarly, if you are working with Markdown, you might want to strip the formatting to get to the underlying text. The strip_markdown function can be used for this purpose, converting **bold** text into just bold.

Removing Irrelevant and Sensitive Content

For many RAG applications, elements like URLs, email addresses, and phone numbers are noise that can distort the semantic meaning of a chunk. Removing them helps the embedding model focus on the primary content. More importantly, removing or redacting Personally Identifiable Information (PII) is a standard practice for privacy and data protection.

You can remove these elements individually or as part of a larger cleaning pipeline. For sensitive applications, instead of just removing PII, you might choose to redact it, replacing the sensitive data with a generic placeholder.

from kerb.safety import redact_pii

sensitive_text = "Please contact support at [email protected] or call 555-123-4567 for help."
redacted_text = redact_pii(sensitive_text)

print(f"Original: {sensitive_text}")
print(f"Redacted: {redacted_text}")
# Original: Please contact support at [email protected] or call 555-123-4567 for help.
# Redacted: Please contact support at [REDACTED_EMAIL] or call [REDACTED_PHONE] for help.

This approach preserves the context that a piece of information was present while protecting the sensitive data itself.

Building a Preprocessing Pipeline

While you can apply these functions one by one, it's more efficient to define a reusable preprocessing pipeline. The normalize_text function, combined with a NormalizationConfig, allows you to orchestrate multiple cleaning and normalization steps in a declarative way. You can define different configurations for different types of content or stages of your RAG pipeline.

Let's build a standard pipeline for processing web-scraped articles. The goal is to lowercase the text, remove URLs and emails, and normalize all whitespace and quotes.

from kerb.preprocessing import normalize_text, NormalizationConfig, NormalizationLevel

# Define a configuration for our pipeline
web_article_config = NormalizationConfig(
    level=NormalizationLevel.STANDARD, # Applies standard unicode, quote, and whitespace normalization
    lowercase=True,
    remove_urls=True,
    remove_emails=True,
    remove_extra_spaces=True
)

messy_article_snippet = """
Check   out our new "Report" at https://example.com/report!
Contact [email protected] for info.
It's an in-depth analysis of AI’s impact.
"""

# Apply the entire pipeline in one call
processed_text = normalize_text(messy_article_snippet, config=web_article_config)

print("### Before Preprocessing ###")
print(messy_article_snippet)
print("\n### After Preprocessing ###")
print(processed_text)

This "before and after" comparison shows the result of our pipeline:

Before Preprocessing

Check   out our new "Report" at https://example.com/report!
Contact [email protected] for info.
It's an in-depth analysis of AI’s impact.

After Preprocessing

check out our new "report" at !
contact for info.
it's an in-depth analysis of ai's impact.

The output is now a clean, standardized block of text. This cleaned version is far more suitable for an embedding model, as the noise has been stripped away, leaving only the core textual content. The level of preprocessing is a balance. Overly aggressive cleaning might remove important context, so it's always good practice to test the impact of your pipeline on retrieval quality.

With our documents loaded, chunked, and now thoroughly cleaned, we have a high-quality dataset of text fragments. Each fragment is a semantically coherent and normalized unit of information, ready for the next step: converting text into numerical representations through embeddings.

Was this section helpful?

References

Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, 2008 (Cambridge University Press) - A classic textbook on information retrieval, detailing how text processing and indexing are fundamental for effective search and retrieval systems.
The Unicode Standard, The Unicode Consortium, 2025 (The Unicode Consortium) - The official standard defining character encoding, including normalization forms (NFC, NFD, NFKC, NFKD) which are important for consistent text processing.