var script = document.createElement("script");
script.src = "https://widget.kapa.ai/kapa-widget.bundle.js";
script.setAttribute("data-website-id", "8ae12a97-484a-4704-8127-b6f17ebc6bcf");
script.setAttribute("data-project-name", "Unstructured");
script.setAttribute("data-project-color", "#0CDDF8");
script.setAttribute(
  "data-project-logo",
  "https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png"
);
document.body.appendChild(script);

Blog

Community

Product

Unstructured

Welcome

Platform

Open Source

Example Code

Glossary

Contact Us

Destination Connectors

**Unstructured** provides tools to ingest and preprocess unstructured documents for `Retrieval Augmented Generation (RAG)` and `Fine Tuning`.

Overview

Unstructured Open Source

Quick Start

Unstructured open source library offers flexible options for installation.

Full Installation

The instructions below guide you on how to use the unstructured library inside a Docker container.

Docker Installation

The `unstructured` library includes functions to partition, chunk, clean, and stage raw source documents. These functions serve as the primary public interfaces within the library.

Core Functionality

Partitioning functions in `unstructured` allow users to extract structured content from a raw unstructured document. These functions break a document down into elements such as `Title`, `NarrativeText`, and `ListItem`, enabling users to decide what content they’d like to keep for their particular application. If you’re training a summarization model, for example, you may only be interested in `NarrativeText`.

Partitioning

As part of data preparation for an NLP model, it’s common to need to clean up your data prior to passing it into the model. If there’s unwanted content in your output, for example, it could impact the quality of your NLP model. To help with this, the `unstructured` library includes cleaning functions to help users sanitize output before sending it to downstream applications.

Cleaning

Extracting

Staging

Chunking functions in `unstructured` use metadata and document elements detected with `partition` functions to post-process elements into more useful "chunks" for uses cases such as Retrieval Augmented Generation (RAG).

Chunking

Embedding encoder classes in `unstructured` use document elements detected with `partition` or document elements grouped with `chunking` to obtain embeddings for each element, for uses cases such as Retrieval Augmented Generation (RAG).

Embedding

The Ingest Library is a powerful tool designed to coordinate the process of pulling data from data providers, partitioning the content, and pushing that new content to a desired location. This technical documentation will provide an in-depth understanding of the Ingest Library, including its features, architecture, installation, configuration, usage, API reference, troubleshooting, examples, and more.

Ingest

Source Connectors

Airtable

Azure

Biomed

Confluence

Delta Table

Discord

Dropbox

Elasticsearch

GitHub

GitLab

Google Cloud Storage

Google Drive

Jira

Local

MongoDB

Notion

OneDrive

OpenSearch

Outlook

Reddit

Salesforce

SFTP

SharePoint

Slack

Wikipedia

Astra

Azure Cognitive Search

Chroma

Clarifai

Databricks Volumes

Google Cloud Service

Pinecone

Qdrant

Vectara

Weaviate

Processor Configuration

Read Configuration

Partition Configuration

Permissions Configuration

Retry Strategy Configuration

Chunking Configuration

Embedding Configuration

Fsspec Configuration

Document elements and metadata

Depending on your need, `Unstructured` provides OCR-based and Transformer-based models to detect elements in the documents. The models are useful to detect the complex layout in the documents and predict the element types.

Models

Partitioning strategies

Integrate your model development pipeline with your favorite machine learning frameworks and libraries, and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of [staging functions](/open-source/core-functionality/staging), which take a list of `Element` objects as input and return formatted dictionaries as output.

Integrations

Unstructured Serverless API Services

This page describes how to obtain an API key to use with the free Unstructured API, the limitations of the free Unstructured API, and provides a quick start example.

Free Unstructured API

This page describes how to get started with the Unstructured Serverless API. Learn how to obtain an API key to use with the Unstructured Serverless API, and get started in no time.

Unstructured Serverless API

Unstructured API on Azure

This guide provides step-by-step instructions for deploying Unstructured API from AWS Marketplace.

Unstructured API on AWS

This page provides some examples of accessing Unstructured API via direct calls.

Sending POST requests to Unstructured API

This documentation covers the usage of the Python and TypeScript SDKs for interacting with the Unstructured API.

Python and TypeScript SDKs

Using API from partition_via_api()

This page provides some examples of accessing Unstructured API via different methods

Examples

The Unstructured API provides parameters to customize the processing of documents. Below are the details for these parameters.

API Parameters

This section details the structure of HTTP validation errors returned by the API.

API Validation Errors

Chunking strategies

Pipeline

Unstructured Platform

Quickstart

SaaS Unstructured Platform Quickstart

Using Workflows

Using Jobs

Source connectors are essential components in data integration systems that establish a link between your files and the data ingestion process. They facilitate the batch processing of files, allowing for the systematic retrieval and ingestion of data stored in various file formats.

Platform Source Connectors

This page contains the information to ingest your documents from Amazon S3 buckets.

Amazon S3

This page contains the information to ingest your documents from Azure Blob Storage.

Azure Blob Storage

This page contains the information to ingest your data from Elasticsearch.

This page contains the information to ingest your data from Google Cloud Storage.

This page contains the information to ingest your data from Google Drive.

This page contains the information to ingest your data from OneDrive.

OneDrive Cloud Storage

This page contains the information to ingest your data from OpenSearch.

This page contains the information to ingest your data from Salesforce.

This page contains the information to ingest your data from an SFTP server.

SFTP Storage

This page contains the information to ingest your documents from Sharepoint sites.

Destination Connectors in the `Unstructured Platform` are designed to specify the endpoint for data processed within the platform. These connectors ensure that the transformed and analyzed data is securely and efficiently transferred to a storage system for future use, often to a vector database for tasks that involve high-speed retrieval and advanced data analytics operations.

Platform Destination Connectors

his page contains the information to store processed data to Amazon S3.

This page contains the information to store processed data to Azure Cognitive Search.

This page contains the information to store processed data to a Chroma instance.

This page contains the information to store processed data to Databricks.

Databricks

This page contains the information to store processed data to an Elasticsearch cluster.

This page contains the information to store processed data to Google Cloud Storage.

This page contains the information to store processed data to a MongoDB database.

This page contains the information to store processed data to an OpenSearch cluster.

This page contains the information to store processed data to Pinecone vector database.

This page contains the information to store processed data to a PostgreSQL database.

Unstructured Open Source

Getting Started With Open Source

Using Unstructured Open Source

Data Ingestion

Concepts

Integrations

Destination Connectors