Chunking

After partitioning, chunking rearranges the resulting document elements into manageable “chunks” to stay within the limits of an embedding model and to improve retrieval precision. The goal is to retrieve only parts of documents that contain only the information that is relevant to a user’s query. You can specify if and how Unstructured chunks those elements, based on your intended end use.

During chunking, Unstructured uses a basic chunking strategy that attempts to combine two or more consecutive text elements into each chunk that fits together within Max characters. To determine the best Max characters length, see the documentation for the embedding model that you want to use.

You can further control this behavior with by-title, by-page, or by-similarity chunking strategies. In all cases, Unstructured will only split individual elements if they exceed the specified Max characters length. After chunking, you will have document elements of only the following types:

• CompositeElement: Any text element will become a CompositeElement after chunking. A composite element can be a combination of two or more original text elements that together fit within the Max characters length. It can also be a single element that doesn’t leave room in the chunk for any others but fits by itself. Or it can be a fragment of an original text element that was too big to fit in one chunk and required splitting.
• Table: A table element is not combined with other elements, and if it fits within Max characters it will remain as is.
• TableChunk: Large tables that exceed Max characters are split into special TableChunk elements.

Here are a few examples:

{
    "type": "CompositeElement",
    "element_id": "3e891e70121efac5f8374e4f1d41f25c",
    "text": "PORTABLE DOCUMENT FORMAT (PDF) SPECIFICATIONS...<full-content-redacted-for-brevity>...",
    "metadata": {
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 1,
      "orig_elements": "eJy...<full-content-redacted-for-brevity>...tY0=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}

{
    "type": "Table",
    "element_id": "2abca97d9f7dde6557cda4731f5e1d01",
    "text": "Date 2005-04-08 2008-06-04 2011-12-20 2012-1-13 2014-9-26 2016-9-14...<full-content-redacted-for-brevity>...",
    "metadata": {
      "text_as_html": "<table><thead><tr><th>Date</th><th>Version</th><th>Summary of Revisions</th></tr></thead><tbody>...<full-content-redacted-for-brevity>...",
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 3,
      "orig_elements": "eJz...<full-content-redacted-for-brevity>...r8=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}

The following sections provide information about the available chunking strategies and their settings.

​

Basic chunking strategy

The basic chunking strategy uses only Max characters and New after n characters to combine sequential elements to maximally fill each chunk. This strategy does not use section boundaries, page boundaries, or content similarities to determine the chunks’ contents.

To use this chunking strategy, choose Chunk by character in the Chunkers section of a Chunker node in a workflow.

​

Chunk by title strategy

The by-title chunking strategy attempts to preserve section boundaries when determining the chunks’ contents. A single chunk should not contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

To use this chunking strategy, choose Chunk by title in the Chunkers section of a Chunker node in a workflow.

​

Chunk by page strategy

The by-page chunking strategy attempts to preserve page boundaries when determining the chunks’ contents. A single chunk should not contain text that occurred in two different page. When a new page starts, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

To use this chunking strategy, choose Chunk by page in the Chunkers section of a Chunker node in a workflow.

​

Chunk By similarity strategy

The by-similarity chunking strategy uses the sentence-transformers/multi-qa-mpnet-base-dot-v1 embedding model to identify topically similar sequential elements and combines them into chunks.

As with the other chunking strategies, chunks will never exceed the absolute maximum chunk size set by Max characters. For this reason, not all elements that share a topic will necessarily appear in the same chunk. However, with this strategy you can guarantee that two elements with low similarity will not be combined in a single chunk.

To use this chunking strategy, choose Chunk by similarity in the Chunkers section of a Chunker node in a workflow.

You can control the level of topic similarity you require for elements to have by setting Similarity threshold.

​

Max characters setting

Specifies the absolute maximum number of characters in a chunk.

To specify this setting, enter a number into the Max characters field.

This setting applies to all of the chunking strategies.

​

Combine text under n characters setting

Combines elements from a section into a chunk until a section reaches a length of this many characters.

To specify this setting, enter a number into the Combine text under n chars field.

This setting applies only to the chunking strategy Chunk by title.

​

Include original elements setting

If the Include original elements box is checked, the elements that were used to form a chunk appear in the metadata field’s orig_elements field for that chunk.

This setting applies to all of the chunking strategies.

​

Multipage sections setting

If the Multipage sections box is checked, this allows sections to span multiple pages.

This setting applies only to the chunking strategy Chunk by title.

​

New after n characters setting

Closes new sections after reaching a length of this many characters. This is an approximate limit.

To specify this setting, enter a number into the New after n characters field.

This setting applies only to the chunking strategies Chunk by character, Chunk by title, and Chunk by page.

​

Overlap setting

Applies a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting.

To specify this setting, enter a number into the Overlap field.

This setting applies only to the chunking strategies Chunk by character, Chunk by title, and Chunk by page.

​

Overlap all setting

If the Overlap all box is checked, applies overlap to “normal” chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units.

This setting applies only to the chunking strategies Chunk by character, Chunk by title, and Chunk by page.

​

Similarity threshold setting

Specifies the minimum similarity that text in consecutive elements must have to be included in the same chunk. This must be a value between 0.0 and 1.0, exclusive (0.01 to 0.99). The default is 0.5 if not otherwise specified.

To specify this setting, enter a number into the Similarity threshold field.

This setting applies only to the chunking strategy Chunk by similarity.

​

Contextual chunking

A technique known as contextual chunking prepends chunk-specific explanatory context to each chunk. Contextual chunking has been shown to enhance traditional RAG solutions by yielding significant improvements in retrieval accuracy, which directly translates to better performance in downstream tasks. Learn more.

To apply contextual chunking, switch on the Contextual chunking toggle in the settings for any chunking strategy.

This chunk-specific explanatory context information is typically a couple of sentences in length. Contextual chunking happens before any embeddings are generated.

When contextual chunking is applied, the contextual information in each chunk begins with Prefix: and ends with a semicolon (;). The chunk’s original content begins with Original:.

For example, without contextual chunking applied, elements would for instance be generated similar to the following. Line breaks have been inserted here for readability. The output will not contain these line breaks:

{
    "type": "CompositeElement",
    "element_id": "aa482034de5ade41b7223bb3beeb6a22",
    "text": "THE\n\nCONSTITUTION of the United States\n\nG\n\nNATIONAL 
      CONSTITUTION CENTER\n\nWe the People of the United States, in 
      Order to form a more perfect Union, establish Justice, insure 
      ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1
    }
},
{
    "type": "CompositeElement",
    "element_id": "59fbfcfb51e52c426df4c48a620c6031",
    "text": "SECTION. 2\n\nThe House of Representatives shall be 
        composed of Mem- bers chosen every second Year by the People 
        of the several States, and the Electors in each State shall 
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 2
    }
},

Applying contextual chunking to those same elements would result in the following output. Line breaks and blank lines have been inserted here for readability. The output will not contain these line breaks and blank lines:

{
    "type": "CompositeElement",
    "element_id": "063ed41d2a989191f2281b2d35c4b4ae",
    "text": "Prefix: This is the opening preamble and first section of 
        Article I of the U.S. Constitution, establishing the fundamental 
        purpose of the document and the basic structure of legislative 
        power in Congress. It appears at the very beginning of the main 
        constitutional text, before all other articles and amendments.;
        
        Original: THE\n\nCONSTITUTION of the United States\n\nG\n\nNATIONAL 
        CONSTITUTION CENTER\n\nWe the People of the United States, in 
        Order to form a more perfect Union, establish Justice, insure 
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1
    }
},
{
    "type": "CompositeElement",
    "element_id": "2270f6b8c8b4afc668f6277789370ffd",
    "text": "Prefix: This chunk appears in Article I, Section 2 of the 
        U.S. Constitution, which establishes the structure, composition, 
        and powers of the House of Representatives as one of the two 
        chambers of Congress. It follows Section 1's establishment of 
        Congress and precedes Section 3's establishment of the Senate.;
        
        Original: SECTION. 2\n\nThe House of Representatives shall be 
        composed of Mem- bers chosen every second Year by the People 
        of the several States, and the Electors in each State shall
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 2
    }
}

​

Learn more

  Chunking for RAG: best practices.