Text Embeddings and Similarity Metrics in Advanced RAG Pipelines

Introduction #

In modern Artificial Intelligence and Natural Language Processing, standalone Large Language Models (LLMs) frequently face limitations when tasked with answering domain-specific queries or reasoning over proprietary corporate documents. Retrieval-Augmented Generation (RAG) bridges this gap by connecting LLMs to external knowledge sources—such as PDFs, documentation sites, code repositories, and vector databases—enabling models to deliver grounded, accurate, and up-to-date responses.

However, feeding raw, massive documents directly into an LLM pipeline leads to severe inefficiencies, inflated API costs, context truncation, and degraded retrieval accuracy. This is where Text Splitters and Chunking Strategies play a pivotal role.

In this comprehensive developer guide, you will learn:

• What text splitters are and why chunking is foundational to high-performance RAG pipelines.
• The 5 core chunking strategies—from character-length splitting to advanced semantic and LLM-based chunking.
• How text splitters operate under the hood within framework architectures like LangChain.
• Practical Python code implementations for Python code, Markdown headers, JSON objects, and semantic text.
• The mathematics of text embeddings and similarity metrics—including Euclidean Distance, Cosine Similarity, and Dot Product.
• Best practices for selecting the optimal chunking strategy for production RAG systems.

What Are RAG Text Splitters & Chunking Strategies? #

Simple Definition #

A text splitter is a software component that takes a large body of text or a long document and breaks it into smaller, logically organized segments called chunks. A chunking strategy is the specific algorithmic rule or heuristic used to decide where those cuts should occur.

Technical Definition #

In a RAG ingestion pipeline, a text splitter transforms raw text strings or standardized Document objects into a sequence of smaller, bounded sub-documents. Each chunk is engineered to stay within predefined token limits while maximizing internal semantic coherence—ensuring that related information remains unified within a single vector embedding.

Real-World Analogy #

Imagine organizing an entire university textbook. If you sliced the book into exact 1,000-word blocks using a paper cutter, you would routinely cut sentences in half and separate core definitions from their supporting diagrams. Instead, a human librarian divides the book naturally by chapters, sections, and sub-headings. Text splitters perform this exact organization programmatically—ensuring every chunk forms a complete, self-contained unit of meaning.

Small Example #

Consider this input paragraph:

“Machine learning models learn patterns from historical data. Deep neural networks extend this by utilizing multiple layers of artificial neurons.”

• Naïve Character Splitting (Size = 50):
  • Chunk 1: "Machine learning models learn patterns from historical "
  • Chunk 2: "data. Deep neural networks extend this by utilizing " (Notice how “historical data” is split across chunks, breaking context).
• Structure-Aware Chunking (Sentence-Based):
  • Chunk 1: "Machine learning models learn patterns from historical data."
  • Chunk 2: "Deep neural networks extend this by utilizing multiple layers of artificial neurons." (Preserves full semantic context in each chunk).

Why Is Text Splitting Essential in RAG? #

Skipping the chunking step or relying on unoptimized chunking degrades RAG performance across five major dimensions:

1. Model Context Window Boundaries #

Every LLM possesses a maximum input context limit (e.g., 8k, 32k, or 128k tokens). Attempting to inject unchunked multi-page documents directly into the prompt prompt risks exceeding context limits, throwing runtime errors, or incurring severe financial costs per call.

2. The “Lost in the Middle” Phenomenon #

Research demonstrates that LLMs pay the highest attention to information located at the very beginning and very end of an input prompt. When presented with massive, unchunked blocks of text, LLMs struggle to recall or reason over facts situated in the middle of the context window. Splitting long documents into focused chunks mitigates this attention degradation.

3. Downstream Retrieval Precision #

If an entire 50-page PDF is converted into a single vector embedding, querying a specific detail (e.g., “What is the warranty policy on page 37?”) forces the system to retrieve the entire document. By breaking the document into 500 distinct chunk vectors, vector databases can perform nearest-neighbor search to retrieve only the 2 or 3 most relevant paragraphs, presenting a highly filtered context to the LLM.

4. Preventing Embedding Compression Loss #

Embedding models encode textual meaning into fixed-dimensional vectors (e.g., 512, 768, or 1536 floats). Compressing 10,000 words into a 512-float vector results in heavy information loss and fuzzy semantic representations. Compressing a focused 500-word chunk into the same 512-float vector preserves fine-grained semantic nuances with minimal compression loss.

5. Efficient Parallel Processing #

Generating embeddings for thousands of individual chunks can be easily batched and computed in parallel across multi-GPU or distributed worker nodes. Processing a single giant monolithic text string cannot be easily parallelized.

Key Concepts in Text Splitting & Embedding #

Position in the RAG Pipeline #

In a standard RAG application architecture, text splitters operate as the second major stage—immediately following Document Loaders and preceding Embedding Generation:

1. Document Loaders: Ingest raw files (PDFs, HTML, GitHub repos) and standardise them into Document objects containing raw page_content and metadata dicts.
2. Text Splitters: Take Document objects or raw strings and partition them into smaller, optimized chunks.
3. Embedding Models: Convert text chunks into dense floating-point vector representations.
4. Vector Databases: Store vectors and metadata for fast similarity search.

Critical Configuration Parameters #

When instantiating text splitters, developers adjust four fundamental parameters:

How Text Splitters Work (Under the Hood) #

In modern AI frameworks like LangChain, all text splitters inherit from a common base class (BaseDocumentTransformer / TextSplitter). This class hierarchy equips every splitter with three standard methods:

The Chunk Overlap Mechanism #

When splitting text strictly by character limits, words or concepts at chunk boundaries risk being severed. Setting a chunk_overlap parameter (typically 10%–20% of chunk_size) copies the tail end of Chunk N into the head of Chunk N+1.

This structural stitch ensures that key phrases spanning boundary cuts are preserved completely in at least one chunk, preventing semantic context breakage during LLM prompt retrieval.

Detailed Explanation of the 5 Chunking Strategies #

Strategy 1: Character Length-Based Chunking #

What Is It? #

Character length-based chunking is the simplest splitting technique. It iterates through a text string and inserts a hard split whenever the character or token count hits chunk_size.

Why Is It Needed? #

It serves as a lightweight baseline for unstructured text without explicit structural formatting or formatting tags.

How Does It Work? #

1. Start at character position 0.
2. Count characters up to chunk_size.
3. Look for the designated separator (e.g., space or empty string).
4. Slice the string and start the next chunk at chunk_size - chunk_overlap.

Practical Implementation #

pip install -U langchain-text-splitters or uv add langchain-text-splitters

from langchain_text_splitters import CharacterTextSplitter

text = """Artificial Intelligence is transforming modern software engineering.
Machine learning algorithms analyze large datasets to uncover hidden patterns.
Deep learning neural networks process complex inputs like images and audio."""

# Character-based splitting
char_splitter = CharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=20,
    separator=" ",
    length_function=len
)

chunks = char_splitter.split_text(text)
print(f"Total Chunks Created: {len(chunks)}")

Token-Based Splitting (TikToken) #

Rather than counting raw Python characters, token-based splitters utilize model-specific tokenizers (e.g., OpenAI’s tiktoken with cl100k_base encoding) to ensure chunks match LLM token constraints precisely:

from langchain_text_splitters import CharacterTextSplitter

token_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",  # GPT-4 / GPT-3.5 tokenizer
    chunk_size=50,                # 50 Tokens maximum
    chunk_overlap=5
)

token_chunks = token_splitter.split_text(text)

Advantages & Disadvantages #

• Advantages: Blazing fast execution speed, predictable memory footprint, minimal CPU overhead.
• Disadvantages: Context-blind; frequently cuts sentences or words in half, damaging semantic coherence.

Strategy 2: Text Structure-Based / Recursive Chunking #

What Is It? #

Recursive Character Text Splitting is the industry standard default for general text in LangChain. It attempts to keep paragraphs, sentences, and words intact by applying a list of separators hierarchically.

Why Is It Needed? #

Single-character splitting destroys paragraph structure and breaks sentences mid-thought. Recursive splitting preserves natural language boundaries.

How Does It Work? #

The algorithm evaluates a pre-defined list of hierarchical separators:

1. "\n\n" (Double newlines / Paragraphs)
2. "\n" (Single newlines / Lines)
3. " " (Spaces / Words)
4. "" (Characters)

It attempts to split on "\n\n" first. If a resulting paragraph chunk exceeds chunk_size, it recursively falls back to "\n" to break that paragraph into sentences. If a sentence is still too long, it falls back to " ". Finally, it runs an internal optimization method (_merge_splits) to combine small adjacent chunks up to chunk_size.

Practical Implementation #

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """Artificial Intelligence Overview.

Machine learning is a subfield of AI focused on building systems that learn from data.
Supervised learning uses labeled datasets to train models.

Deep learning utilizes multi-layer neural networks for computer vision and NLP."""

recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=150,
    chunk_overlap=20,
    separators=["\n\n", "\n", " ", ""]
)

chunks = recursive_splitter.split_text(text)
for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ({len(chunk)} chars) ---")
    print(chunk)

Advantages & Disadvantages #

• Advantages: Excellent balance between chunk size adherence and semantic coherence; preserves sentence integrity.
• Disadvantages: Variable chunk lengths; requires tuned separator lists for specialized document formats.

Strategy 3: Document Structure-Based Chunking #

What Is It? #

Document structure-based chunking leverages the inherent syntax and hierarchy of structured file formats—such as HTML tags, Markdown headers, Python/Java source code, or JSON objects.

Why Is It Needed? #

Applying standard character splitting to source code or structured files breaks classes, functions, and JSON keys, rendering code unparseable and losing structural context.

Implementation A: Python Source Code Splitting #

Uses RecursiveCharacterTextSplitter.from_language with language-specific key terms (e.g., class, def, tab indentations):

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

python_code = """
import numpy as np

class DataAnalyzer:
    def __init__(self, data):
        self.data = data

    def calculate_mean(self):
        return np.mean(self.data)

    def calculate_std(self):
        return np.std(self.data)
"""

code_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=200,
    chunk_overlap=0
)

code_chunks = code_splitter.split_text(python_code)

Implementation B: Markdown Header Splitting #

MarkdownHeaderTextSplitter splits text at specified Markdown headers (#, ##, ###) and automatically injects the header hierarchy directly into the chunk’s metadata dict:

from langchain_text_splitters import MarkdownHeaderTextSplitter

markdown_text = """
# Artificial Intelligence

## Machine Learning
Machine learning algorithms build mathematical models based on sample data.

### Supervised Learning
Supervised learning involves learning a function that maps inputs to output targets.

## Deep Learning
Deep learning models use complex representations at higher layers.
"""

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False
)

doc_chunks = markdown_splitter.split_text(markdown_text)
# Output chunks will feature metadata like: {'Header 1': 'Artificial Intelligence', 'Header 2': 'Machine Learning'}

Implementation C: JSON Splitting #

RecursiveJsonSplitter splits nested JSON documents while preserving key-value structural relationships:

from langchain_text_splitters import RecursiveJsonSplitter

json_data = {
    "company": "AI Research Corp",
    "departments": [
        {"name": "Machine Learning", "projects": ["LLM Fine-tuning", "RAG Systems"]},
        {"name": "Data Engineering", "projects": ["ETL Pipelines", "Vector Stores"]}
    ]
}

json_splitter = RecursiveJsonSplitter(max_chunk_size=100)
json_chunks = json_splitter.split_json(json_data)

Advantages & Disadvantages #

• Advantages: Respects syntactic boundaries; enriches chunk metadata with structural headers or keys.
• Disadvantages: Only applicable to structured file formats; can produce large chunks if a single function or JSON object is very long.

Strategy 4: Semantic Similarity-Based Chunking #

What Is It? #

Semantic Chunking evaluates the actual mathematical meaning of text. Rather than splitting on character thresholds, it measures semantic distance between adjacent sentences and inserts splits only when semantic meaning shifts significantly.

Why Is It Needed? #

Character and recursive splitters cannot detect topic transitions occurring inside continuous paragraphs. Semantic chunking creates variable-sized chunks aligned strictly with distinct topics.

How Does It Work? #

1. Split the raw document into individual sentences.
2. Generate dense vector embeddings for each sentence using an embedding model (e.g., OpenAI embeddings).
3. Calculate Cosine Similarity between consecutive sentence vectors.
4. Compare similarity drops against a breakpoint_threshold_type:
   • percentile: Splits when similarity drops below a target percentile (e.g., 95th percentile).
   • standard_deviation: Splits when similarity drops by N standard deviations from the mean.
   • interquartile: Uses IQR outlier detection for extreme topic shifts.
5. Group continuous, semantically similar sentences into unified chunks.

Practical Implementation #

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

text = """Artificial Intelligence and deep neural networks have revolutionized natural language processing.
Transformer architectures enable massive parallelization during training.

Italian pasta recipes require high quality durum wheat semolina and fresh eggs.
Boil salted water thoroughly before adding linguine.

Global climate change continues to increase sea surface temperatures worldwide."""

# Initialize Semantic Chunker with OpenAI Embeddings
semantic_chunker = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=90.0
)

semantic_chunks = semantic_chunker.split_text(text)

Advantages & Disadvantages #

• Advantages: High semantic coherence; follows natural topic boundaries perfectly.
• Disadvantages: High computational cost and latency (requires embedding calls for every sentence); variable and unpredictable chunk sizes.

Strategy 5: LLM-Based Chunking #

What Is It? #

LLM-Based Chunking utilizes a Large Language Model equipped with Natural Language Understanding (NLU) to inspect raw text, detect topic boundaries, and output structured chunks alongside metadata summaries.

Why Is It Needed? #

Complex, unstructured text with implicit topic shifts can defeat rule-based and similarity-threshold splitters. LLMs act as intelligent technical editors.

How Does It Work? #

1. Pass a text passage to an LLM with a system prompt instructing it to split text at natural topic transitions.
2. Enforce a structured output format using Pydantic schemas (BaseModel).
3. Ask the LLM to generate a concise 1-2 sentence summary for each chunk.
4. Store the chunk text in page_content and the summary in metadata.

Raw Text --> LLM Prompt + Pydantic Schema --> [Chunk 1 Text + Summary 1, Chunk 2 Text + Summary 2]

Practical Implementation #

from pydantic import BaseModel, Field
from typing import List
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.documents import Document

# 1. Define Pydantic Schema for Structured Output
class SingleChunk(BaseModel):
    chunk_text: str = Field(description="The exact extracted chunk text.")
    summary: str = Field(description="A 1-2 sentence summary of this chunk.")

class ChunkerResponse(BaseModel):
    chunks: List[SingleChunk]

# 2. Setup LLM with Structured Output
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_llm = llm.with_structured_output(ChunkerResponse)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are an expert text chunker. Split the text at natural topic boundaries without altering wording. Provide a brief summary for each chunk."),
    ("human", "Split this text:\n\n{input_text}")
])

chain = prompt | structured_llm

# 3. Execute LLM Chunking
text_content = "..." # Raw input text
response = chain.invoke({"input_text": text_content})

# 4. Convert to LangChain Document Objects
doc_objects = [
    Document(page_content=c.chunk_text, metadata={"summary": c.summary})
    for c in response.chunks
]

Advantages & Disadvantages #

• Advantages: Highly intelligent; handles ambiguous text seamlessly; automatically enriches metadata with chunk-level summaries.
• Disadvantages: Most expensive strategy (API token costs); slowest processing speed; non-deterministic output.

Deep Dive: Embeddings & Similarity Metrics in Retrieval #

The Evolution of Text Representation #

To understand how vector retrieval operates on generated chunks, we must review the three evolutionary stages of text vectorization:

Stage 1: Classical (BoW / TF-IDF)  High-dim, Sparse, Context-Blind
Stage 2: Static Embeddings (Word2Vec) Low-dim, Dense, Context-Blind
Stage 3: Contextual (BERT / OpenAI)  Low-dim, Dense, Context-Aware

1. Stage 1: Classical Methods (Bag of Words / TF-IDF): Created high-dimensional, sparse vectors equal to vocabulary size (10,000+ dimensions). Almost all values were zero, leading to computationally expensive matrix operations and complete lack of context.
2. Stage 2: Static Word Embeddings (Word2Vec / GloVe): Introduced compact, dense vectors (~768 dimensions). However, they were context-blind—assigning the exact same vector to the word “bank” in “river bank” and “money bank”.
3. Stage 3: Contextual Embeddings (BERT / OpenAI / Gemma): Modern standard. Produces dense, context-aware vectors where a word’s representation dynamically adapts based on surrounding text.

The Embedding Space (Hyper-space) #

In an N-dimensional vector space (e.g., 512 or 1536 dimensions), contextually similar chunks group together into semantic clusters. A query vector placed in this space sits near its most relevant chunk neighbors.

Comparison of Similarity & Distance Metrics #

Architecture & Complete RAG Workflow #

Best Practices & Summary #

1. Default to Recursive Chunking First: Start with RecursiveCharacterTextSplitter (chunk_size=500-1000, chunk_overlap=100-200) for standard text.
2. Match Splitters to Document Types: Use MarkdownHeaderTextSplitter for technical docs, RecursiveCharacterTextSplitter.from_language for code, and RecursiveJsonSplitter for API payloads.
3. Set Overlap to 10%–20%: Prevent boundary context loss by configuring consistent chunk overlap.
4. Normalize Embeddings for Production: Normalize vector outputs to L2 unit length and utilize Dot Product search in vector stores to minimize query retrieval latency.
5. Reserve Semantic & LLM Chunking for High-Value Domains: Use SemanticChunker or LLM-based chunking selectively when document quality is paramount and API budget permits.

Text Embeddings Quiz #