---
name: Unstructured
description: Use when processing unstructured documents (PDFs, images, emails, etc.) into structured JSON data for RAG applications, AI agents, and data pipelines. Use Unstructured Transform for local files and quick prototyping, or Unstructured Pipelines for large-scale remote document processing with connectors to 35+ sources and destinations.
metadata:
    mintlify-proj: unstructured
    version: "1.0"
---

# Unstructured Skill

## Product summary

Unstructured converts unstructured files (PDFs, images, emails, Word docs, spreadsheets, etc.) into clean, structured JSON data ready for AI applications, RAG systems, and data pipelines. It offers two main services: **Transform** (for local files, quick prototyping, and AI tools via MCP) and **Pipelines** (for large-scale remote document processing with 35+ connectors). Core operations include partitioning (document-to-JSON), extraction (custom schema), chunking (for embeddings), enrichment (image/table descriptions), and embedding (vector generation). Access via Python SDK, REST API, or AI tool integrations (Claude, Cursor, VS Code, etc.).

**Key resources:**
- Primary docs: https://docs.unstructured.io
- Python SDK: `unstructured` package
- REST API: Unstructured API endpoints (Transform or Pipelines)
- CLI/MCP: Unstructured Transform MCP server for AI tools
- Config: Workflow YAML/JSON for Pipelines; Python/cURL for Transform

## When to use

Reach for Unstructured when:

- **Processing documents for RAG**: Partition PDFs, images, emails, and other files into structured elements with metadata (coordinates, parent-child relationships, language detection)
- **Extracting structured data**: Define a JSON schema and extract specific fields from documents (invoices, forms, contracts)
- **Building AI pipelines**: Use Transform MCP with Claude, Cursor, or other AI tools to process files with plain-language prompts
- **Scaling document processing**: Use Pipelines to batch-process files from S3, Google Drive, Dropbox, Salesforce, etc., and write results to vector DBs (Pinecone, Weaviate), data warehouses (Snowflake), or blob storage
- **Chunking and embedding**: Prepare partitioned output for embedding models by splitting into semantic chunks with configurable strategies
- **Enriching documents**: Add AI-generated summaries for images and tables, convert tables to HTML, perform NER, or apply generative OCR
- **Choosing partitioning strategies**: Balance speed, cost, and quality—use Auto (recommended), Fast (text-only), High Res (advanced extraction), or VLM (handwriting, multilanguage)

## Quick reference

### Partitioning strategies

| Strategy | Best for | Speed | Cost | Quality |
|----------|----------|-------|------|---------|
| **Auto** | Mixed content, unknown file types | Medium | Medium | High (routes per-page) |
| **Fast** | Text-only documents | Very fast | Low | Medium |
| **High Res** | Complex layouts, images, tables | Slow | Medium | High |
| **VLM** | Handwriting, multilanguage, scanned docs | Slowest | High | Highest |

### Chunking strategies

| Strategy | Use case | Preserves |
|----------|----------|-----------|
| **By Character** | Generic content, no structure | None (character limits only) |
| **By Title** | Documents with headings/sections | Section boundaries |
| **By Page** | Page-based documents | Page boundaries |
| **By Similarity** | Topic-coherent chunks | Semantic similarity (0.01–0.99 threshold) |
| **Contextual** | RAG with improved retrieval | Adds LLM-generated context prefix to chunks |

### Supported file types

60+ formats: PDF, DOCX, PPTX, XLSX, CSV, HTML, TXT, MD, EML, MSG, PNG, JPEG, TIFF, HEIC, and more. See `api-reference/supported-file-types` for complete list.

### Core operations (in order)

1. **Partition**: File → structured JSON elements (Title, NarrativeText, Table, Image, etc.)
2. **Extract** (optional): Define schema, extract custom fields into JSON
3. **Chunk** (optional): Split partitioned elements into embedding-sized pieces
4. **Enrich** (optional): Add image descriptions, table summaries, NER, generative OCR
5. **Embed** (optional): Generate vectors using OpenAI, Hugging Face, or other providers
6. **Persist**: Write to vector DB, data warehouse, or blob storage

### Transform vs. Pipelines

| Feature | Transform | Pipelines |
|---------|-----------|-----------|
| **Input** | Local files | Remote sources (S3, Drive, Dropbox, etc.) |
| **Scale** | Single files or small batches | Large-scale batch processing |
| **UI** | AI tool integration (MCP) or web | No-code web dashboard |
| **Connectors** | None (local only) | 35+ sources, 30+ destinations |
| **Pricing** | $0.015/page after 10k free pages | Usage-based, contact sales |
| **Setup** | Minutes (MCP or API) | Requires Business account |

## Decision guidance

### When to use Transform vs. Pipelines

**Use Transform if:**
- Processing local files or small batches
- Prototyping or testing
- Using AI tools (Claude, Cursor, VS Code)
- Need quick results without infrastructure
- Free tier sufficient (10k pages/month)

**Use Pipelines if:**
- Processing 100s or 1000s of files at scale
- Files stored in cloud (S3, Google Drive, Dropbox, Salesforce)
- Need scheduled/automated processing
- Require connectors to vector DBs or data warehouses
- Enterprise compliance needed (SOC 2, HIPAA, GDPR, FedRAMP)

### When to use each partitioning strategy

**Auto (recommended):** Mixed documents, unknown types, cost-conscious. Routes each page to optimal strategy.

**Fast:** Text-only PDFs, emails, plain text. Skips images/tables. ~100x faster than VLM.

**High Res:** Complex layouts, images, tables, bounding box coordinates needed. Slower, medium cost.

**VLM:** Handwritten text, multilanguage (Japanese, Arabic, etc.), scanned documents, highest quality. Slowest, highest cost.

### When to use each chunking strategy

**By Character:** No document structure, generic content, simple size limits.

**By Title:** Documents with clear heading hierarchy (reports, manuals, articles).

**By Page:** Page-based documents (books, PDFs with page breaks).

**By Similarity:** Topic-coherent content, semantic relevance important for retrieval.

**Contextual:** RAG systems where chunk context improves retrieval accuracy (add after other chunkers).

## Workflow

### Typical Transform workflow (local files)

1. **Prepare file**: Ensure file is supported format (PDF, DOCX, PNG, etc.)
2. **Choose tool**: Use Claude, Cursor, VS Code, or Python SDK
3. **Connect MCP** (if using AI tool): Add Unstructured Transform connector
4. **Upload file**: Drag-drop or use file manager
5. **Describe intent**: "Partition this PDF" or "Extract invoice fields using this schema"
6. **Download results**: JSON output with partitioned elements or extracted data
7. **Verify**: Check element types, metadata, coordinates, extracted fields

### Typical Pipelines workflow (remote files)

1. **Create source connector**: Connect to S3, Google Drive, Dropbox, etc. (requires credentials)
2. **Create destination connector**: Connect to Pinecone, Weaviate, Snowflake, S3, etc.
3. **Build workflow**: Add nodes in DAG:
   - Source (auto-created)
   - Partitioner (choose strategy: Auto, Fast, High Res, VLM)
   - Chunker (choose strategy, set max_characters, overlap)
   - Enricher (optional: image descriptions, table summaries, NER)
   - Embedder (optional: choose provider/model)
   - Destination (auto-created)
4. **Test workflow**: Run on sample file, inspect output
5. **Schedule or run**: Manual run or set schedule (cron)
6. **Monitor jobs**: Check status, view logs, download output
7. **Verify results**: Confirm data in destination (vector DB, warehouse, etc.)

### Typical API workflow (Python SDK)

1. **Install SDK**: `pip install unstructured-client`
2. **Get credentials**: API key and URL from Transform or Pipelines dashboard
3. **Create job** (Transform): `client.jobs.create_job(files=[...], partition_by="auto", ...)`
4. **Poll for completion**: `client.jobs.get_job(job_id)` until status is "SUCCEEDED"
5. **Download results**: `client.jobs.download_job_output(job_id, file_id)`
6. **Parse JSON**: Load results, iterate over elements
7. **Verify**: Check element types, metadata, extracted fields

## Common gotchas

- **Partitioning strategy mismatch**: Using Fast on image-heavy PDFs loses images. Use Auto or VLM for mixed content.
- **Chunk size too small**: Chunks <128 tokens lose context. Chunks >1024 tokens cause information overload. Experiment with 256–512 tokens.
- **Missing max_characters for embeddings**: If using embeddings, set chunker's max_characters ≤ embedding model's recommended limit (e.g., 512 for OpenAI).
- **Overlapping chunks**: Overlap repeats characters from previous chunk; can introduce noise. Use sparingly (25–50 chars).
- **API key/URL mismatch**: Each workspace has unique API key and URL. Verify in dashboard (Transform or Pipelines).
- **Unsupported file types**: Check `api-reference/supported-file-types`. Encrypted/password-protected PDFs fail.
- **VLM quota limits**: Large PDFs (200+ pages) with VLM may hit quota errors. Use High Res or Fast as fallback.
- **Contextual chunking placement**: Must be added *after* text chunker (character, title, page, similarity), not before.
- **Connector auth failures**: Verify credentials, permissions, and network access. Test connection before running jobs.
- **Enrichment without partitioning**: Image descriptions, table summaries only work if partitioner is Auto or High Res (not Fast).

## Verification checklist

Before submitting work:

- [ ] File format is supported (check `api-reference/supported-file-types`)
- [ ] Partitioning strategy matches content type (Auto for mixed, Fast for text-only, VLM for handwriting)
- [ ] Chunking strategy aligns with document structure (By Title for headings, By Page for page breaks, etc.)
- [ ] max_characters set appropriately for embedding model (≤ model's recommended limit)
- [ ] API key and URL are correct and match workspace
- [ ] Source and destination connectors tested (if using Pipelines)
- [ ] Workflow DAG is valid (source → partitioner → [chunker] → [enricher] → [embedder] → destination)
- [ ] Sample output reviewed (element types, metadata, extracted fields correct)
- [ ] Enrichment nodes placed after partitioner, contextual chunker after text chunker
- [ ] No encrypted/password-protected files in batch
- [ ] Scheduling/retry logic configured (if using Pipelines)

## Resources

- **Comprehensive navigation**: https://docs.unstructured.io/llms.txt
- **Partitioning strategies**: https://docs.unstructured.io/concepts/partitioning
- **Chunking strategies**: https://docs.unstructured.io/concepts/chunking
- **Transform quickstart**: https://docs.unstructured.io/transform/get-started/overview
- **Pipelines quickstart**: https://docs.unstructured.io/pipelines/quickstart
- **API reference**: https://docs.unstructured.io/api-reference/api/overview
- **Python SDK**: https://docs.unstructured.io/overview-python-sdk
- **Troubleshooting**: https://docs.unstructured.io/support/issues/overview

---

> For additional documentation and navigation, see: https://docs.unstructured.io/llms.txt