Retrieval-Augmented Generation, commonly known as RAG, allows a large language model to answer questions using information from external sources such as company documents, policies, manuals, knowledge bases, and databases.

However, before documents can be stored and retrieved effectively, they usually need to be divided into smaller pieces called chunks.

Chunking may appear to be a simple preprocessing step, but it has a major impact on the quality of a RAG system. If chunks are too large, they may contain irrelevant information and consume unnecessary context-window space. If they are too small, important ideas may be separated, leaving the model without enough context to produce an accurate answer.

Two commonly used approaches are fixed-size chunking and semantic splitting. Each has advantages, limitations, and suitable use cases.

What Is Chunking in RAG?

A RAG pipeline typically converts documents into smaller text segments before generating vector embeddings for them. Each chunk is stored as a separate searchable record, usually with metadata connecting it to the original document.

When a user submits a question, the system compares the question with the stored chunks and retrieves those considered most relevant. The selected chunks are then provided to the language model as supporting context.

Therefore, the quality of the generated answer depends heavily on whether the retrieved chunks contain complete and relevant information. Microsoft describes chunking as one of the first major optimization points in a RAG data pipeline because chunking decisions directly affect the information retrieved for the model.

What Is Fixed-Size Chunking?

Fixed-size chunking divides text according to a predefined number of characters, words, sentences, or tokens.

For example, a developer may configure a system to create one chunk for every 500 tokens. A small overlap, such as 50 tokens, may be included between neighbouring chunks to preserve some continuity.

Suppose a document contains 2,000 tokens. Using a 500-token chunk size could produce approximately four or five chunks, depending on the overlap settings.

The main advantage of this approach is simplicity. It is easy to implement, predictable, fast, and relatively inexpensive. It works well when processing large volumes of content or documents that do not have a reliable structure, such as application logs, raw transcripts, exported data, or long blocks of plain text.

Fixed-size chunking also makes infrastructure planning easier because most chunks have similar lengths. This can simplify embedding generation, storage estimation, retrieval configuration, and performance testing.

However, fixed boundaries do not understand meaning. A chunk may end in the middle of a sentence, explanation, procedure, or argument. Microsoft notes that fixed-size windows can separate words, sentences, or paragraphs, potentially disrupting comprehension and information flow.

Why Chunk Overlap Is Used

Chunk overlap is commonly added to reduce the risk of losing context at a boundary.

For example, if one chunk ends with the beginning of a product-return procedure, the next chunk may repeat the final few sentences from the previous one. This gives the retrieval system more opportunities to find a complete answer.

Overlap can improve continuity, but it also creates duplicate information. More overlapping content means more embeddings, greater storage requirements, and potentially repetitive retrieval results. The overlap should therefore be large enough to preserve context but not so large that it unnecessarily expands the index.

Microsoft’s Azure AI Search documentation recommends using a small amount of overlap when applying fixed-size chunking to help maintain continuity between neighbouring segments.

What Is Semantic Splitting?

Semantic splitting divides content according to meaning rather than a rigid length.

Instead of cutting a document every 500 tokens, a semantic chunker may detect changes in subject, paragraph structure, headings, sentence relationships, or embedding similarity. Content discussing one idea remains together, while a meaningful topic change creates a new chunk.

For example, an employee handbook may contain separate sections covering attendance, leave, workplace conduct, and information security. A semantic splitter would try to keep each policy section intact rather than dividing the text at arbitrary token boundaries.

This generally produces more coherent chunks. When a chunk represents one complete concept, it can be easier for the retrieval system to match it with a user’s question. The language model also receives cleaner context with fewer incomplete statements.

Semantic chunking is especially useful for policy documents, research papers, legal content, technical manuals, educational material, and other documents where meaning is closely connected to structure.

Limitations of Semantic Splitting

Semantic splitting is usually more complex and computationally expensive than fixed-size chunking.

Some implementations analyse sentence similarity using embeddings, while others use language models or document-intelligence systems to identify logical boundaries. This adds processing time and cost during document ingestion.

Chunk sizes may also vary significantly. One topic could produce a short chunk, while another may produce a section too large for the embedding model or retrieval pipeline. Semantic splitting therefore still requires minimum and maximum size controls.

It can also struggle with poorly formatted documents, scanned files, tables, mixed layouts, or content where topic boundaries are unclear.

Fixed-Size vs. Semantic Splitting

Fixed-size chunking is generally the better starting point when simplicity, speed, scalability, and predictable processing are priorities. It is also suitable for early prototypes because developers can quickly experiment with chunk size and overlap.

Semantic splitting is often more appropriate when preserving complete ideas is critical. It can improve retrieval quality for structured or knowledge-heavy documents, although it requires more processing and careful implementation.

In practice, the strongest approach is often hybrid chunking.

A hybrid strategy may first split a document using headings, sections, or paragraphs and then apply a maximum token limit to oversized sections. Small neighbouring paragraphs may also be combined until they reach a useful minimum size.

This approach preserves document meaning while keeping chunks compatible with embedding and language-model limits.

How to Choose the Right Strategy

There is no universal chunk size or splitting method that works for every RAG application. The right choice depends on document structure, expected user questions, embedding-model limits, retrieval settings, and the level of detail required in answers.

Teams should test multiple configurations using real questions. Useful measurements include retrieval precision, answer correctness, context relevance, duplicate retrieval, latency, and token consumption.

A chunk should ideally be understandable on its own, focused on one topic, and detailed enough to answer a relevant question.

Final Thoughts

Chunking is not merely about dividing text into smaller sections. It is about creating searchable units that preserve the knowledge needed to answer users accurately.

Fixed-size chunking offers speed, consistency, and simplicity. Semantic splitting provides stronger contextual coherence and better preservation of meaning. Neither strategy is automatically superior.

For many projects, fixed-size chunking with overlap is a sensible baseline. As the RAG system matures, semantic or hybrid splitting can be introduced where document complexity demands it.

The best strategy is the one that consistently retrieves the smallest amount of content required to provide a complete, accurate, and grounded answer.