This page was recently updated. What do you think about it? Let us know!.

Connect Confluence to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your spaces and pages and store structured outputs locally on your filesystem.

The requirements are as follows.

The following video provides related setup information for Confluence Cloud:

The Confluence connector dependencies:

CLI, Python
pip install "unstructured-ingest[confluence]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

  • CONFLUENCE_URL - The target Confluence site’s URL, represented by --url (CLI) or url (Python).

  • One of the following:

    • For API token authentication: CONFLUENCE_USERNAME and CONFLUENCE_TOKEN - The name or email address,and API token of the target Confluence user, represented by --username (CLI) or username (Python) and --token (CLI) or token (Python), respectively.
    • For personal access token (PAT) authentication: CONFLUENCE_TOKEN - The PAT for the target Confluence user, represented by --token (CLI) or token (Python).
    • For password authentication: CONFLUENCE_USERNAME and CONFLUENCE_PASSWORD - The name or email address, and password of the target Confluence user, represented by --username (CLI) or username (Python) and --password (CLI) or password (Python), respectively.

Additional settings include:

  • --spaces (CLI) or spaces (Python): Optionally, the list of the names of the specific spaces to access, expressed as a comma-separated list of strings (CLI) or an array of strings (Python), with each string representing a space’s name. The default is no specific spaces, if not otherwise specified.
  • --max-num-of-spaces (CLI) or max_num_of_spaces (Python): Optionally, the maximum number of spaces to access, expressed as an integer. The default value is 500 if not otherwise specified.
  • --max-num-of-docs-from-each-space (CLI) or max_num_of_docs_from_each_space (Python): Optionally, the maximum number of documents to access from each space, expressed as an integer. The default value is 100 if not otherwise specified.
  • --cloud or --no-cloud (CLI) or cloud (Python): Optionally, whether to use Confluence Cloud (--cloud for CLI or cloud=True for Python). The default is --no-cloud (CLI) or cloud=False (Python) if not otherwise specified.

These environment variables:

  • UNSTRUCTURED_API_KEY - Your Unstructured API key value.
  • UNSTRUCTURED_API_URL - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

#!/usr/bin/env bash

# For API token authentication:
unstructured-ingest \
  confluence \
    --token $CONFLUENCE_TOKEN \
    --url $CONFLUENCE_URL \
    --username $CONFLUENCE_USERNAME \
    --cloud \
    --spaces luke,paul \
    --max-num-of-spaces 500 \
    --max-num-of-docs-from-each-space 150 \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --chunking-strategy by_title \
    --embedding-provider huggingface

# For personal access token (PAT) authentication:
unstructured-ingest \
  confluence \
    --token $CONFLUENCE_TOKEN \
    --url $CONFLUENCE_URL \
    --spaces luke,paul \
    --max-num-of-spaces 500 \
    --max-num-of-docs-from-each-space 150 \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --chunking-strategy by_title \
    --embedding-provider huggingface

# For password authentication:
unstructured-ingest \
  confluence \
    --password $CONFLUENCE_PASSWORD \
    --url $CONFLUENCE_URL \
    --username $CONFLUENCE_USERNAME \
    --no-cloud \
    --spaces luke,paul \
    --max-num-of-spaces 500 \
    --max-num-of-docs-from-each-space 150 \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --chunking-strategy by_title \
    --embedding-provider huggingface