Chunking functions in unstructured
use metadata and document elements detected with partition
functions to post-process elements into more useful “chunks” for uses cases such as retrieval-augmented generation (RAG).
unstructured
differs from other chunking mechanisms you may be familiar with. Typical approaches start with the text extracted from the document and form chunks based on plain-text features, character sequences like "\n\n"
or "\n"
that might indicate a paragraph boundary or list-item boundary.
Because unstructured
uses specific knowledge about each document format to partition the document into semantic units (document elements), we only need to resort to text-splitting when a single element exceeds the desired maximum chunk size. Except in that case, all chunks contain one or more whole elements, preserving the coherence of semantic units established during partitioning.
A few concepts about chunking are worth introducing before discussing the details.
CompositeElement
, Table
, or TableChunk
elements. Each “chunk” is an instance of one of these three types.
by_title
) may have additional options.
max_characters: int (default=500)
- the hard maximum size for a chunk. No chunk will exceed this number of characters. A single element that by itself exceeds this size will be divided into two or more chunks using text-splitting.
new_after_n_chars: int (default=max_characters)
- the “soft” maximum size for a chunk. A chunk that already exceeds this number of characters will not be extended, even if the next element would fit without exceeding the specified hard maximum. This can be used in conjunction with max_characters
to set a “preferred” size, like “I prefer chunks of around 1000 characters, but I’d rather have a chunk of 1500 (max_characters) than resort to text-splitting”. This would be specified with (..., max_characters=1500, new_after_n_chars=1000)
.
overlap: int (default=0)
- only when using text-splitting to break up an oversized chunk, include this number of characters from the end of the prior chunk as a prefix on the next. This can mitigate the effect of splitting the semantic unit represented by the oversized element at an arbitrary position based on text length.
overlap_all: bool (default=False)
- also apply overlap between “normal” chunks, not just when text-splitting to break up an oversized element. Because normal chunks are formed from whole elements that each have a clean semantic boundary, this option may “pollute” normal chunks. You’ll need to decide based on your use-case whether this option is right for you.
chunking_strategy
argument. The current options are basic
and by_title
(described below).
by_title
strategy shares most behaviors with
the basic strategy so we’ll describe the baseline strategy first:
max_characters
(hard-max) and new_after_n_chars
(soft-max) option values.
Table
element is always isolated and never combined with another element. A Table
can be oversized, like any other text element, and in that case is divided into two or more TableChunk
elements using text-splitting.
overlap
is applied between chunks formed by splitting oversized elements and is also applied between other chunks when overlap_all
is True
.
by_title
chunking strategy preserves section boundaries and optionally page boundaries as well. “Preserving” here means that a single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one started, even if the next element would fit in the prior chunk.
In addition to the behaviors of the basic
strategy above, the by_title
strategy has the following behaviors:
Title
element is considered to start a new section. When a Title
element is encountered, the prior chunk is closed and a new chunk started, even if the Title
element would fit in the prior chunk.
multipage_sections
argument. This defaults to True
meaning that a page break does not start a new chunk. Setting this to False
will separate elements that occur on different pages into distinct chunks.
Title
element even though it does not serve as a section heading. This can produce chunks substantially smaller than desired. This behavior can be mitigated using the combine_text_under_n_chars
argument. This defaults to the same value as max_characters
such that sequential small sections are combined to maximally fill the chunking window. Setting this to 0
will disable section combining.
.metadata.orig_elements
is compressed into Base64 gzipped format. To deserialize .metadata.orig_elements
, you can use the elements_from_base64_gzipped_json
. For example: