Chunking For Rag Maximize Enterprise Knowledge Retrieval

North Mini Code. Cohere's first model for developers.

Learn more

Oct 30, 2024

6 minutes read

Chunking for RAG: Maximize enterprise knowledge retrieval

Learn how to approach some of the most common challenges around chunking – a vital skill to tap enterprise knowledge and succeed with RAG solutions.

!Blog Post Featured Image

As enterprises tap into their vast knowledge bases to leverage generative AI, mastering the critical skill of chunking has become essential for optimizing retrieval-augmented generation (RAG) systems.

Below we answer some of the most common questions our team gets from customers about chunking. Let’s dig in.

##### What is chunking and why is it so important?

Chunking is the process of breaking down large pieces of information into more manageable and context-rich chunks. This makes it easier for AI systems to process, search, and retrieve the most relevant information.

Essentially, chunking organizes information into smaller, more semantically coherent units, improving the AI model’s ability to locate and retrieve the most relevant answers. Chunking typically takes place during pre-processing, where large documents are broken down into smaller sections, such as paragraphs or sentences.

Chunking works by improving the quality of embeddings, the numerical representations of data that a GenAI model can understand and use. When each chunk focuses on a specific piece of information, it makes it easier for the embedding model to create more accurate and detailed numerical representations of the data stored in vector databases. High-quality embeddings can make the difference between a RAG system that delivers on or exceeds expectations and one that falls short.

##### When should you consider chunking?

Chunking is a valuable step for any organization that needs to extract and process information from large documents and data stores as part of a GenAI solution. Breaking content into smaller, more manageable sections improves both the relevance and accuracy of retrieval results.

Because chunking results in fewer parts of documents being sent to the LLM, it also reduces costs and latency. The success of chunking within RAG systems lies in finding the right balance between chunk size and retrieval precision.

##### What size context window should I use?

The context window is the maximum amount of text a model can process at a given time. Context windows are measured by tokens, the building blocks of text such as a word or symbol, and most large language models feature an upper limit.

Chunks are created to fit within the context window of a model. Setting the right context window for a particular use case is crucial to maximize the effectiveness of RAG systems, playing a key role in determining the accuracy and relevance of results, and a model’s overall performance. Set it too small and the chunks won’t have enough semantic connections with the rest of the document for the model to understand its meaning. Too big and the model won’t have enough specificity to retrieve the most relevant and accurate information with which to generate responses.

Limiting the size of the context window and then adjusting it based on retrieval performance is the recommended approach. Cohere Embed, for example, has a maximum chunk size of 512 tokens. If the documents you are embedding have paragraph lengths of around 200 words, for example, a chunk size of 250 words might be appropriate. This overlap, or “sliding window” approach, helps ensure you are collecting some context from the previous and subsequent text. In general, a smaller context window (100-300 words) is better in order for the model to more accurately capture the semantic meaning of text. Cohere’s research and experiments have found that performance is broadly optimized with a few hundred tokens and drops significantly after around 500.

##### What are the different text chunking methods?

For most companies, a chunking strategy will need to include a range of different data types. For unstructured text, here are some common methods we’ve seen to be robust for enterprise customers:

Fixed Size. Content is divided into predetermined chunk sizes. For example you might choose to chunk all documents into 512 token segments. This approach is simple and cost-effective, but it can lose context and is not suitable for smaller documents.

Sentence Level. Text is broken down into individual sentences, allowing for a more granular analysis. This provides a detailed understanding of each sentence's semantics without context from the rest of the text. For this reason, it is ideal for shorter queries or when the focus is on specific sentences rather than broader themes.

Paragraph Level. This method retains more contextual information than sentence-level chunking. It is more suited to longer queries that span more than one sentence or a paragraph, as it is likely looking for broader semantic understanding. This method doesn't require overlap with previous chunks for semantics.

Sliding Window. This technique uses a 'window' of text that slides across an entire document, capturing and processing a set number of consecutive sentences. Unlike paragraph-level chunking, this often creates an overlap with previous chunks to ensure semantic coherence. This is an effective method for processing longer queries as it maintains a broader context to capture the semantic meaning and relationships within the document.

Recursive Chunking. Text is split up into smaller pieces at natural breaks, such as the end of sentences or paragraphs, rather than cutting it at fixed intervals. This makes the text easier to read and understand, and preserves the meaning. But it is more complex to set up and can be tricky to manage due to the creation of chunks of different sizes.

##### How do you chunk structured documents like tables?

Structured document formats, such as tables in PDF files, can pose a different chunking challenge. Each value in a table row or column may be meaningless without its associated headers or neighboring entries. Failing to chunk a large table correctly will lead to a loss of semantic connection and ultimately retrieval results that are out of...