---
name: Unstructured
description: Use when processing unstructured documents (PDFs, images, Word docs, emails, etc.) into AI-ready structured data. Agents should reach for this skill when building document processing pipelines, extracting data from files, creating RAG workflows, or setting up batch processing jobs for document transformation.
metadata:
    mintlify-proj: unstructured
    version: "1.0"
---

# Unstructured Skill

## Product summary

Unstructured transforms unstructured documents (PDFs, images, Word docs, emails, spreadsheets, and 50+ file types) into AI-ready structured data. It provides three main products: **Unstructured Pipelines** (no-code UI for workflows), **Unstructured API** (programmatic access via Python SDK or REST), and **Unstructured Foundation** (persistent file indexing for AI tools). The core processing pipeline includes partitioning (extracting elements), chunking (organizing for retrieval), enriching (adding AI-generated metadata), and embedding (creating vectors). Key file paths: API keys stored in environment variables (`UNSTRUCTURED_API_KEY`, `UNSTRUCTURED_API_URL`). Primary docs: https://docs.unstructured.io

## When to use

Use Unstructured when:
- Building document processing pipelines that ingest files from cloud storage (S3, Google Drive, SharePoint, Dropbox, etc.) and output to vector databases or data warehouses
- Extracting structured data from recurring document types (invoices, forms, contracts) into JSON schemas
- Creating RAG (retrieval-augmented generation) workflows that require chunked, embedded documents
- Processing batches of local files and need to partition, chunk, and enrich them
- Building AI applications that need persistent access to indexed documents across multiple tools
- Handling complex documents (PDFs with images, handwritten text, dense tables) that require vision-language model processing
- Setting up scheduled jobs to continuously process new documents as they arrive

Do not use Unstructured for: simple text parsing, real-time streaming (use batch jobs instead), or documents already in structured formats.

## Quick reference

### Core Processing Steps

| Step | Purpose | Key Options |
|------|---------|------------|
| **Partitioning** | Extract elements from raw files | Auto (default), VLM (highest quality), High Res (all formats), Fast (text-only) |
| **Chunking** | Reorganize elements for retrieval | By character, by title, by page, by similarity |
| **Enriching** | Add AI-generated metadata | Image descriptions, table descriptions, NER, generative OCR |
| **Embedding** | Create vector representations | OpenAI, Anthropic, Hugging Face, or custom models |

### Supported File Types

PDFs, images (.png, .jpg, .heic, .bmp, .tiff), Word (.doc, .docx), Excel (.xls, .xlsx), PowerPoint (.ppt, .pptx), HTML, XML, Markdown, emails (.eml, .msg), EPUB, CSV, TSV, and 30+ more.

### API Endpoints (Pipeline API)

| Task | Method | Path |
|------|--------|------|
| Create workflow | POST | `/workflows` |
| Run workflow | POST | `/workflows/{id}/run` |
| List jobs | GET | `/jobs` |
| Get job status | GET | `/jobs/{id}/details` |
| Create source connector | POST | `/sources` |
| Create destination connector | POST | `/destinations` |

### Environment Variables

```bash
UNSTRUCTURED_API_KEY=your_api_key
UNSTRUCTURED_API_URL=https://api.unstructured.io  # or your custom URL
```

### Chunking Configuration

| Parameter | Default | Purpose |
|-----------|---------|---------|
| `chunk_max_characters` | 500 | Hard limit on chunk size |
| `chunk_new_after_n_chars` | — | Soft limit; start new chunk after this |
| `chunk_overlap` | 0 | Characters to repeat between chunks |
| `chunking_strategy` | None | `basic`, `by_title`, `by_page`, `by_similarity` |

## Decision guidance

### When to use each partitioning strategy

| Strategy | Best for | Speed | Cost | Quality |
|----------|----------|-------|------|---------|
| **Auto** | Mixed document types, production use | Medium | Medium | Highest (adaptive) |
| **VLM** | Images, complex layouts, handwriting | Slow | High | Highest |
| **High Res** | All file types, bounding boxes needed | Medium | Medium | High |
| **Fast** | Text-only documents, speed critical | Very fast | Low | Medium |

### When to use each chunking strategy

| Strategy | Use when | Preserves |
|----------|----------|-----------|
| **By character** | Simple size-based splitting needed | None (pure size-based) |
| **By title** | Documents have clear sections/headings | Section boundaries, hierarchy |
| **By page** | Page boundaries matter (e.g., scanned docs) | Page breaks |
| **By similarity** | Semantic coherence is critical | Topic/semantic boundaries |

### Pipelines vs API vs Foundation

| Aspect | Unstructured Pipelines | Unstructured API | Foundation |
|--------|----------------------|------------------|-----------|
| **Interface** | No-code UI | Python SDK / REST | MCP server |
| **Input** | Remote sources + local files | Local files only | Connected sources |
| **Output** | Vector DBs, data warehouses | JSON (local) | Persistent index |
| **Use case** | Production workflows, scheduling | Quick transforms, testing | AI tool integration |
| **Setup** | Visual workflow builder | Code/curl | Source connections |

## Workflow

### 1. Set up credentials and environment
- Obtain API key and API URL from your Unstructured account
- Set `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL` environment variables
- For Pipelines UI: log in to https://platform.unstructured.io

### 2. Choose your approach
- **For no-code workflows**: Use Unstructured Pipelines UI
- **For programmatic control**: Use Unstructured API with Python SDK or curl
- **For local files only**: Use Unstructured Transform API
- **For persistent AI tool access**: Use Foundation with MCP server

### 3. Create source and destination connectors (Pipelines/API only)
- Click **Connectors** → **Sources** → **New Connector** (or use API)
- Select your source type (S3, Google Drive, Dropbox, etc.)
- Fill in authentication details and test connection
- Repeat for destination (Pinecone, Weaviate, S3, Snowflake, etc.)

### 4. Build or configure workflow
- **UI**: Click **Workflows** → **New Workflow** → drag nodes (Source → Partitioner → Chunker → Embedder → Destination)
- **API**: POST to `/workflows` with source_id, destination_id, and workflow_nodes array
- Configure partitioning strategy (Auto recommended for mixed types)
- Set chunking parameters (max_characters, strategy)
- Add enrichment nodes if needed (image descriptions, NER, table extraction)
- Add embedder node if building RAG (select model: OpenAI, Anthropic, etc.)

### 5. Test and run
- **UI**: Click **Run** next to workflow name
- **API**: POST to `/workflows/{id}/run` or create a job with local files
- Monitor progress in **Jobs** tab
- Check **Job Details** for status, file counts, and any failures

### 6. Verify output
- Download results from job or check destination system
- Inspect sample elements to confirm partitioning quality
- Verify chunks are appropriately sized
- Check embeddings are present if embedding was enabled

## Common gotchas

- **Wrong partitioning strategy for document type**: Using Fast strategy on image-heavy PDFs will miss content. Use Auto or VLM for mixed/complex documents.
- **Chunk size too small**: Default 500 characters may be too small for semantic search. Increase `chunk_max_characters` to 1000–2000 for better context.
- **Missing API credentials**: Ensure both `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL` are set. API URL is not the same as the web UI URL.
- **Workflow stuck in progress**: Previous job still running. Check job status before running again. Adjust schedule frequency if jobs take longer than schedule interval.
- **Embeddings not generated**: Verify embedder node is added to workflow and embedding model is selected. Check that destination supports embeddings.
- **Table extraction failing**: Enable "Infer Table Structure" in connector settings. Some table formats may require High Res or VLM strategy.
- **Large file timeouts**: Use Fast strategy or split large PDFs. Enable `split_pdf_page` for PDFs over 50 pages.
- **Metadata missing**: Metadata is only populated if source connector supports it. Check connector documentation for available fields.
- **Overlapping chunks not working**: `chunk_overlap` only applies to oversized elements split by text. Use `chunk_overlap_all: true` to apply overlap to all chunks.
- **Schema validation errors**: Ensure JSON schema for structured extraction conforms to OpenAI Structured Outputs (max 10 nesting levels).

## Verification checklist

Before submitting work:

- [ ] API credentials are set and environment variables are correct
- [ ] Source connector is created and connection test passes
- [ ] Destination connector is created and connection test passes
- [ ] Partitioning strategy matches document type (Auto for mixed, VLM for images, Fast for text-only)
- [ ] Chunking parameters are appropriate for use case (max_characters, strategy)
- [ ] Enrichment nodes are added if needed (image descriptions, NER, etc.)
- [ ] Embedder node is configured with correct model if building RAG
- [ ] Workflow is enabled (Status toggle is on)
- [ ] Test run completed successfully with sample files
- [ ] Job shows "Finished" status with green checkmark
- [ ] Sample output verified: elements have correct types, metadata is present, chunks are reasonable size
- [ ] No failed files, or failed files are expected (e.g., unsupported formats)
- [ ] Destination system contains processed data (check vector DB, S3, etc.)

## Resources

- **Comprehensive navigation**: https://docs.unstructured.io/llms.txt
- **API Reference**: https://docs.unstructured.io/api-reference/overview
- **Pipelines UI Guide**: https://docs.unstructured.io/pipelines/overview
- **Concepts (partitioning, chunking, enriching)**: https://docs.unstructured.io/concepts/overview
- **Open Source Library**: https://docs.unstructured.io/open-source/introduction/overview

---

> For additional documentation and navigation, see: https://docs.unstructured.io/llms.txt