var script = document.createElement("script");
script.src = "https://widget.kapa.ai/kapa-widget.bundle.js";
script.setAttribute("data-website-id", "8ae12a97-484a-4704-8127-b6f17ebc6bcf");
script.setAttribute("data-project-name", "Unstructured");
script.setAttribute("data-project-color", "#0CDDF8");
script.setAttribute(
  "data-project-logo",
  "https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png"
);
document.body.appendChild(script);

Supported File Types

Data Ingestion

Billing

Get Support

Support

Community

Feedback

Blog

Product

Unstructured

Welcome

Platform

Open Source

Example Code

Ingestion

Glossary

Overview

Unstructured Serverless API Services

Unstructured provides a platform and tools to ingest and process unstructured documents for Retrieval Augmented Generation (RAG) and model fine-tuning.

Unstructured Open Source

Quickstart

Unstructured open source library offers flexible options for installation.

Full Installation

The instructions below guide you on how to use the unstructured library inside a Docker container.

Docker Installation

The `unstructured` library includes functions to partition, chunk, clean, and stage raw source documents. These functions serve as the primary public interfaces within the library.

Core Functionality

Partitioning functions in `unstructured` allow users to extract structured content from a raw unstructured document. These functions break a document down into elements such as `Title`, `NarrativeText`, and `ListItem`, enabling users to decide what content they’d like to keep for their particular application. If you’re training a summarization model, for example, you may only be interested in `NarrativeText`.

Partitioning

As part of data preparation for an NLP model, it’s common to need to clean up your data prior to passing it into the model. If there’s unwanted content in your output, for example, it could impact the quality of your NLP model. To help with this, the `unstructured` library includes cleaning functions to help users sanitize output before sending it to downstream applications.

Cleaning

Extracting

Staging

Chunking functions in `unstructured` use metadata and document elements detected with `partition` functions to post-process elements into more useful "chunks" for uses cases such as Retrieval Augmented Generation (RAG).

Chunking

Embedding encoder classes in `unstructured` use document elements detected with `partition` or document elements grouped with `chunking` to obtain embeddings for each element, for uses cases such as Retrieval Augmented Generation (RAG).

Embedding

The Ingest Library is a powerful tool designed to coordinate the process of pulling data from data providers, partitioning the content, and pushing that new content to a desired location. This technical documentation will provide an in-depth understanding of the Ingest Library, including its features, architecture, installation, configuration, usage, API reference, troubleshooting, examples, and more.

Ingest

Source Connectors

Airtable

Astra

Azure

Biomed

Confluence

Delta Table

Discord

Dropbox

Elasticsearch

GitHub

GitLab

Google Cloud Storage

Google Drive

Jira

Local

MongoDB

Notion

OneDrive

OpenSearch

Outlook

Reddit

Salesforce

SFTP

SharePoint

Slack

Wikipedia

Destination Connectors

Azure Cognitive Search

Chroma

Clarifai

Databricks Volumes

Google Cloud Service

Pinecone

Qdrant

SingleStore

Vectara

Weaviate

Processor Configuration

Read Configuration

Partition Configuration

Permissions Configuration

Retry Strategy Configuration

Chunking Configuration

Embedding Configuration

Fsspec Configuration

Set the OCR agent

Document elements and metadata

Depending on your need, `Unstructured` provides OCR-based and Transformer-based models to detect elements in the documents. The models are useful to detect the complex layout in the documents and predict the element types.

Models

Partitioning strategies

Integrate your model development pipeline with your favorite machine learning frameworks and libraries, and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of [staging functions](/open-source/core-functionality/staging), which take a list of `Element` objects as input and return formatted dictionaries as output.

Integrations

This page describes how to get started with the Unstructured Serverless API. Learn how to obtain an API key to use with the Unstructured Serverless API, and get started in no time.

Unstructured Serverless API

This page describes how to obtain an API key to use with the free Unstructured API, the limitations of the free Unstructured API, and provides a quickstart example.

Free Unstructured API

Unstructured API on Azure

Follow these steps to deploy the Unstructured API service into your AWS account.

Unstructured API on AWS

This page provides some examples of accessing Unstructured API via direct calls.

Sending POST requests to Unstructured API

This documentation covers the usage of the Python and JavaScript/TypeScript SDKs for interacting with the Unstructured API.

Python and JavaScript/TypeScript SDKs

Using API from partition_via_api()

This page provides some examples of accessing Unstructured API via different methods.

Examples

The Unstructured API provides parameters to customize the processing of documents. Below are the details for these parameters.

API Parameters

This section details the structure of HTTP validation errors returned by the API.

API Validation Errors

Parse simple PDFs and HTML

Set partitioning behavior

Set chunking behavior

Output unique element IDs

Output bounding box coordinates

Set document language for better OCR

Extract tables as HTML

Extract image block types

Change element coordinate systems

Work with PowerPoint files

Use LangChain and Ollama

Use LangChain and Llama 3

Notebooks contain complete working sample code for end to end solutions.

Notebooks

Chunking strategies

Pipeline

Unstructured Platform

Unstructured Platform Quickstart

Using Workflows

Using Jobs

Source connectors are essential components in data integration systems that establish a link between your files and the data ingestion process. They facilitate the batch processing of files, allowing for the systematic retrieval and ingestion of data stored in various file formats.

Platform Source Connectors

This page contains the information to ingest your documents from Amazon S3 buckets.

Amazon S3

This page contains the information to ingest your documents from Azure Blob Storage.

Azure Blob Storage

This page contains the information to ingest your data from Elasticsearch.

This page contains the information to ingest your data from Google Cloud Storage.

This page contains the information to ingest your data from Google Drive.

This page contains the information to ingest your data from OneDrive.

OneDrive Cloud Storage

This page contains the information to ingest your data from OpenSearch.

This page contains the information to ingest your data from Salesforce.

This page contains the information to ingest your data from an SFTP server.

SFTP Storage

This page contains the information to ingest your documents from Sharepoint sites.

Destination Connectors in the `Unstructured Platform` are designed to specify the endpoint for data processed within the platform. These connectors ensure that the transformed and analyzed data is securely and efficiently transferred to a storage system for future use, often to a vector database for tasks that involve high-speed retrieval and advanced data analytics operations.

Platform Destination Connectors

his page contains the information to store processed data to Amazon S3.

This page contains the information to store processed data to Azure Cognitive Search.

This page contains the information to store processed data to a Chroma instance.

This page contains the information to store processed data to Databricks.

Databricks

This page contains the information to store processed data to an Elasticsearch cluster.

This page contains the information to store processed data to Google Cloud Storage.

This page contains the information to store processed data to a MongoDB database.

This page contains the information to store processed data to an OpenSearch cluster.

This page contains the information to store processed data to Pinecone vector database.

This page contains the information to store processed data to a PostgreSQL database.

Category	File Types
Plaintext	.eml, .html, .md, .msg, .rst, .rtf, .txt, .xml
Images	.png, .jpg, .jpeg, .tiff, .bmp, .heic
Documents	.csv, .doc, .docx, .epub, .odt, .pdf, .ppt, .pptx, .tsv, .xlsx

Unstructured API Services

Getting Started With API Services

Using Unstructured API

Data Ingestion

How To

Best Practices

Concepts

Endpoints

Unstructured Serverless API Services

Unstructured Serverless API

Free Unstructured API

Unstructured API on Azure

Unstructured API on AWS

Supported File Types

Data Ingestion

Billing

Get Support

Unstructured API Services

Getting Started With API Services

Using Unstructured API

Data Ingestion

How To

Best Practices

Concepts

Endpoints

Unstructured Serverless API

Free Unstructured API

Unstructured API on Azure

Unstructured API on AWS

​Supported File Types

​Data Ingestion

​Billing

​Get Support

Supported File Types

Data Ingestion

Billing

Get Support