In the journey of building RAG (Retrieval-Augmented Generation) applications, after loading your documents, the next critical step is Text Splitters. This guide, based on the CampusX series, explores how to break down large documents into manageable “chunks” to ensure your LLM performs effectively and accurately.
1. What is Text Splitting? #
Text Splitting is the process of breaking large pieces of text—such as articles, PDFs, or books—into smaller, manageable pieces called chunks.
The code that performs this operation is called a Text Splitter. In a standard RAG workflow, you connect a Document Loader to a Text Splitter, which then outputs a list of smaller Document objects.
2. Why Do We Need Text Splitters? #
There are three major reasons why you should never feed an entire large document into an LLM at once:
- Context Length Limits: Every LLM has a maximum input size (e.g., 50,000 tokens). If your PDF has 100,000 words, you will breach the model’s threshold and fail to process the document.
- Output Quality: LLMs perform better with smaller, focused contexts. Large texts can cause the model to “drift” or even hallucinate information that isn’t in the document.
- Better Embeddings & Search: It is difficult to capture the semantic meaning of a huge document in a single vector. Breaking text into chunks (e.g., one chunk per IPL team) allows embedding models to capture specific meanings more precisely, leading to more accurate Semantic Search.
- Resource Optimization: Smaller chunks are more memory-efficient and allow for better parallelization during processing.
3. Key Concepts: Size and Overlap #
When configuring a splitter, you will interact with two main parameters:
- chunk_size: The maximum number of units (characters or tokens) in each chunk.
- chunk_overlap: The number of characters shared between two consecutive chunks. This is crucial for retaining context that might otherwise be lost if a sentence or word is cut off mid-way.
- Best Practice: For RAG applications, an overlap of 10% to 20% of the chunk size is generally recommended.
4. Types of Text Splitters #
A. Length-Based (CharacterTextSplitter)
The simplest method. It traverses the text and creates a split once the chunk_size is reached, regardless of the text’s structure.
- Pros: Very fast and simple.
- Cons: Often cuts off text in the middle of words or sentences, losing semantic meaning.
from langchain_text_splitters import CharacterTextSplitter
splitter = CharacterTextSplitter(
chunk_size=100,
chunk_overlap=20,
separator=""
)
chunks = splitter.split_text(your_text)
B. Text Structure-Based (RecursiveCharacterTextSplitter)
This is the most recommended and widely used splitter. It tries to keep related pieces of text together by splitting based on a hierarchy of separators:
- Double Newlines (Paragraphs)
- Single Newlines (Sentences)
- Spaces (Words)
- Characters
It only moves down the hierarchy if a chunk is still too large, ensuring your text is rarely cut mid-word or mid-sentence.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=300,
chunk_overlap=30
)
chunks = splitter.split_documents(docs) # docs from a Document Loader
C. Specialized Document Splitters (Code & Markdown)
For non-plain text documents like Python code or Markdown, standard paragraph splitting doesn’t work. These splitters use specific keywords (like class or def in Python) to split logic intelligently.
# For Python Code
splitter = RecursiveCharacterTextSplitter.from_language(
language="python",
chunk_size=300,
chunk_overlap=0
)
D. Semantic Meaning-Based (SemanticChunker)
This is an experimental approach that doesn’t use length or structure. Instead, it uses Embeddings to compare consecutive sentences.
- How it works: It calculates the similarity between sentences. When the similarity drops significantly (using a threshold like Standard Deviation), it identifies a “topic change” and performs a split.
- Status: Currently experimental and may not always yield satisfying or accurate results compared to recursive splitting.
5. Summary Table
| Splitter Type | Best For | Logic |
|---|---|---|
| Character | Simple, fast tasks | Strict character count. |
| Recursive | Standard RAG | Hierarchy: Para > Sentence > Word. |
| Code/Markdown | Programming files | Language-specific keywords. |
| Semantic | Topic-based splitting | Embedding similarity (Experimental). |
By mastering these splitters, you ensure that your LLM receives the most relevant, context-rich, and accurately sized information for every query
Text Splitters Quiz #
Q.1 What is the primary objective of using a 'Text Splitter' in the context of Large Language Model (LLM) applications?
To remove stop words and perform lemmatization on raw text data.
To break down large documents into smaller, manageable chunks that an LLM can process effectively.
To compress text data to reduce the storage space required in a vector database.
To translate large documents into multiple languages simultaneously.
Explanation
Text Splitters divide large documents into smaller chunks so they fit within an LLM’s context window and can be processed efficiently while preserving important information.
Q.2 Why is text splitting considered crucial for improving the quality of text embeddings?
Large documents automatically crash embedding models regardless of their content.
It is easier to capture the precise semantic meaning of a small chunk than a very large, multi-topic document.
It eliminates the need for using an embedding model entirely.
It allows the embedding model to generate higher-dimensional vectors.
Explanation
Embedding models create one vector for the entire input. Smaller, focused chunks produce embeddings that better represent a specific topic or concept, improving retrieval quality.
Q.3 An LLM has a context length limit of 50,000 tokens. If you attempt to summarize a PDF containing 100,000 tokens without splitting, what is the most likely outcome?
The LLM will automatically compress the 100,000 tokens into 50,000.
The LLM will process the entire document but take twice as long.
The input will exceed the threshold, likely resulting in an error or truncated information.
The LLM will ignore the limit if the hardware has enough RAM.
Explanation
Every LLM has a fixed context window. If the input exceeds this limit, the request will either fail or the excess text will be truncated, causing information loss.
Q.4 What is the primary purpose of the 'chunk overlap' parameter in text splitting?
To create duplicate data to ensure the LLM doesn't forget the text.
To encrypt the boundaries between chunks for security purposes.
To maintain semantic context between adjacent chunks and prevent abrupt cuts in the middle of sentences.
To reduce the total number of chunks generated from a document.
Explanation
Chunk overlap repeats a small portion of text between neighbouring chunks, helping preserve context and reducing the chance of splitting important ideas across chunk boundaries.
Q.5 Which of the following best describes the logic used by the 'Recursive Character Text Splitter'?
It splits text only based on a fixed character count regardless of punctuation.
It uses a hierarchy of separators like paragraphs (\n\n), lines (\n), and spaces to keep related text together.
It only works for programming languages and cannot process plain English.
It randomly selects split points to ensure a diverse distribution of data.
Explanation
Recursive Character Text Splitter attempts to split using larger separators such as paragraphs first, then lines, spaces, and finally characters if necessary, preserving as much semantic structure as possible.
Q.6 In LangChain, if you are working with a Python source file, how should you initialize a recursive splitter for the best results?
Use the standard splitter with the separator set to 'Python'.
Use the from_language method with Language.PYTHON to use code-specific separators like class and def.
Code cannot be split and must be fed to the LLM as a single chunk.
Manually insert split markers in the code before processing.
Explanation
LangChain provides RecursiveCharacterTextSplitter.from_language(Language.PYTHON), which uses Python-aware separators such as class and def to create more meaningful code chunks.
Q.7 How does 'Semantic Meaning-based' text splitting differ from 'Length-based' splitting?
It uses embeddings to detect when the topic of the text changes and splits at those context shifts.
It only works on documents that have been translated into multiple languages.
It is the fastest and most simple method available in LangChain.
It uses character counts but only for specific fonts.
Explanation
Semantic text splitting analyses the meaning of neighbouring sentences using embeddings and creates chunk boundaries where significant topic changes occur instead of relying only on character or token counts.
Q.8 What role does the 'Standard Deviation' play in the experimental Semantic Chunker?
It serves as a threshold to determine if the 'distance' between two sentences is large enough to warrant a split.
It calculates the average length of all words in the document.
It determines the speed at which the LLM generates tokens.
It is used to count the number of grammatical errors in a chunk.
Explanation
The Semantic Chunker compares embedding distances between neighbouring sentences and uses a standard deviation threshold to decide when a semantic change is significant enough to split the text.
Q.9 According to the source, what is a recommended chunk overlap percentage for RAG-based applications?
50% to 75%
10% to 20%
Exactly 100%
0% to 5%
Explanation
A chunk overlap of approximately 10% to 20% is commonly recommended for RAG applications because it preserves context while avoiding excessive duplication of information.
Q.10 True or False: The 'Character Text Splitter' always ensures that sentences are never cut in the middle.
False
True
Explanation
Character Text Splitter uses character counts rather than sentence boundaries, so it may split text in the middle of a sentence. Recursive or semantic splitters are better choices when preserving sentence structure is important.
