> ## Documentation Index
> Fetch the complete documentation index at: https://docs.unstructured.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Confluence
Connect Confluence to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your spaces and pages and store structured outputs locally on your filesystem.
The requirements are as follows.
* A [Confluence Cloud account](https://www.atlassian.com/software/confluence/pricing) or
[Confluence Data Center installation](https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html).
* The site URL for your [Confluence Cloud account](https://community.atlassian.com/t5/Confluence-questions/confluence-cloud-url/qaq-p/1157148) or
[Confluence Data Center installation](https://confluence.atlassian.com/confkb/how-to-find-your-site-url-to-set-up-the-confluence-data-center-and-server-mobile-app-938025792.html).
* A user in your [Confluence Cloud account](https://confluence.atlassian.com/cloud/invite-edit-and-remove-users-744721624.html) or
[Confluence Data Center installation](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html).
* The user must have the correct permissions in your
[Conflunce Cloud account](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) or
[Confluence Data Center installation](https://confluence.atlassian.com/doc/permissions-and-restrictions-139557.html) to
access the target spaces and pages.
* One of the following:
* For Confluence Cloud or Confluence Data Center, the target user's name or email address, and password.
[Change a Confluence Cloud user's password](https://support.atlassian.com/confluence-cloud/docs/change-your-confluence-password/).
[Change a Confluence Data Center user's password](https://confluence.atlassian.com/doc/change-your-password-139416.html).
* For Confluence Cloud only, the target user's name or email address, and API token.
[Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
* For Confluence Data Center only, the target user's personal access token (PAT).
[Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).
* Optionally, the keys (not display names) of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access. To get a space's key,
which is different from a space's display name, open the space in your web browser and look at the URL. The space's key is the part of the URL after `spaces/` but before the next `/` after that.
The following video provides related setup information for Confluence Cloud:
## Document permissions metadata
The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.
The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.
```json theme={null}
[
{
"...": "...",
"metadata": {
"...": "...",
"data_source": {
"...": "...",
"permissions_data": [
{
"read": {
"users": [
"11111:11111111-1111-1111-1111-111111111111"
],
"groups": [
"22222222-2222-2222-2222-22222222",
"33333333-3333-3333-3333-33333333"
]
}
},
{
"update": {
"users": [
"44444:44444444-4444-4444-4444-44444444",
"55555:55555555-5555-5555-5555-55555555"
],
"groups": [
"66666666-6666-6666-6666-66666666",
]
}
},
{
"delete": {
"users": [
"77777:77777777-7777-7777-7777-77777777"
],
"groups": [
"88888888-8888-8888-8888-88888888"
]
}
}
],
"...": "..."
}
}
}
]
```
To look up information about a particular Confluence user, use the user's ID (also known as their *account ID*) from the preceding output to call the
[GET /wiki/rest/api/user](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-users/#api-wiki-rest-api-user-get)
operation in the Confluence REST API.
To look up information about a particular Confluence group, use the group's ID from the preceding output to call the
[GET /wiki/rest/api/group/by-id](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-group/#api-wiki-rest-api-group-by-id-get)
operation in the Confluence REST API.
The Confluence connector dependencies:
```bash CLI, Python theme={null}
pip install "unstructured-ingest[confluence]"
```
You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).
The following environment variables:
* `CONFLUENCE_URL` - The target Confluence site's URL, represented by `--url` (CLI) or `url` (Python).
* One of the following:
* For API token authentication: `CONFLUENCE_USERNAME` and `CONFLUENCE_API_TOKEN` - The name or email address, and API token of the target Confluence user, represented by `--username` (CLI) or `username` (Python) and `--api-token` (CLI) or `api-token` (Python), respectively.
* For personal access token (PAT) authentication: `CONFLUENCE_PERSONAL_ACCESS_TOKEN` - The PAT for the target Confluence user, represented by `--token` (CLI) or `token` (Python).
* For password authentication: `CONFLUENCE_USERNAME` and `CONFLUENCE_PASSWORD` - The name or email address, and password of the target Confluence user, represented by `--username` (CLI) or `username` (Python) and `--password` (CLI) or `password` (Python), respectively.
Additional settings include:
* `--spaces` (CLI) or `spaces` (Python): Optionally, the list of the keys (not display names) of the specific spaces to access, expressed as a comma-separated list of strings (CLI) or an array of strings (Python), with each string representing a space's key (not display name). The default is no specific spaces, if not otherwise specified.
* `--max-num-of-spaces` (CLI) or `max_num_of_spaces` (Python): Optionally, the maximum number of spaces to access, expressed as an integer. The default value is `500` if not otherwise specified.
* `--max-num-of-docs-from-each-space` (CLI) or `max_num_of_docs_from_each_space` (Python): Optionally, the maximum number of documents to access from each space, expressed as an integer. The default value is `100` if not otherwise specified.
* `--cloud` or `--no-cloud` (CLI) or `cloud` (Python): Optionally, whether to use Confluence Cloud (`--cloud` for CLI or `cloud=True` for Python). The default is `--no-cloud` (CLI) or `cloud=False` (Python) if not otherwise specified.
* `--extract-images` (CLI) or `extract_images` (Python): Optionally, download images and replace the HTML content with Base64-encoded images. The default is `--no-extract-images` (CLI) or `extract_images=False` (Python) if not otherwise specified.
* `--extract-files` (CLI) or `extract_files` (Python): Optionally, download any embedded files. The default is `--no-extract-files` (CLI) or `extract_files=False` (Python) if not otherwise specified.
* `--force-download` (CLI) or `force_download` (Python): Optionally, re-download extracted files even if they already exist locally. The default is `--no-force-download` (CLI) or `force_download=False` (Python) if not otherwise specified.
* `--allow-list` (CLI) or `allow_list`: Optionally, a comma-separated list (CLI) or an array of strings (Python) of allowed URLs to download. By default, the base URL that the original HTML came from is used, if not otherwise specified.
Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.
This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.
```bash CLI theme={null}
#!/usr/bin/env bash
# For API token authentication:
unstructured-ingest \
confluence \
--api-token $CONFLUENCE_API_TOKEN \
--url $CONFLUENCE_URL \
--username $CONFLUENCE_USERNAME \
--cloud \
--spaces , \
--max-num-of-spaces 500 \
--max-num-of-docs-from-each-space 150 \
--extract-images \
--extract-files \
--force-download \
--output-dir $LOCAL_FILE_OUTPUT_DIR \
--partition-by-api \
--api-key $UNSTRUCTURED_API_KEY \
--partition-endpoint $UNSTRUCTURED_API_URL \
--chunking-strategy by_title \
--embedding-provider huggingface
# For personal access token (PAT) authentication:
unstructured-ingest \
confluence \
--token $CONFLUENCE_PERSONAL_ACCESS_TOKEN \
--url $CONFLUENCE_URL \
--spaces , \
--max-num-of-spaces 500 \
--max-num-of-docs-from-each-space 150 \
--extract-images \
--extract-files \
--force-download \
--output-dir $LOCAL_FILE_OUTPUT_DIR \
--partition-by-api \
--api-key $UNSTRUCTURED_API_KEY \
--partition-endpoint $UNSTRUCTURED_API_URL \
--chunking-strategy by_title \
--embedding-provider huggingface
# For password authentication:
unstructured-ingest \
confluence \
--password $CONFLUENCE_PASSWORD \
--url $CONFLUENCE_URL \
--username $CONFLUENCE_USERNAME \
--no-cloud \
--spaces luke,paul \
--max-num-of-spaces 500 \
--max-num-of-docs-from-each-space 150 \
--extract-images \
--extract-files \
--force-download \
--output-dir $LOCAL_FILE_OUTPUT_DIR \
--partition-by-api \
--api-key $UNSTRUCTURED_API_KEY \
--partition-endpoint $UNSTRUCTURED_API_URL \
--chunking-strategy by_title \
--embedding-provider huggingface
```
```python Python Ingest theme={null}
import os
from unstructured_ingest.pipeline.pipeline import Pipeline
from unstructured_ingest.interfaces import ProcessorConfig
from unstructured_ingest.processes.connectors.confluence import (
ConfluenceIndexerConfig,
ConfluenceDownloaderConfig,
ConfluenceConnectionConfig,
ConfluenceAccessConfig
)
# Chunking and embedding are optional.
from unstructured_ingest.processes.partitioner import PartitionerConfig
from unstructured_ingest.processes.chunker import ChunkerConfig
from unstructured_ingest.processes.embedder import EmbedderConfig
from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
if __name__ == "__main__":
Pipeline.from_configs(
context=ProcessorConfig(),
indexer_config=ConfluenceIndexerConfig(
spaces=["", ""],
max_num_of_spaces=500,
max_num_of_docs_from_each_space=150
),
downloader_config=ConfluenceDownloaderConfig(
download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
extract_images=True,
extract_files=True,
force_download=True,
allow_list=[]
),
# For API token authentication:
source_connection_config=ConfluenceConnectionConfig(
access_config=ConfluenceAccessConfig(
api_token=os.getenv("CONFLUENCE_API_TOKEN")
),
url=os.getenv("CONFLUENCE_URL"),
username=os.getenv("CONFLUENCE_USERNAME"),
cloud=True
),
# For personal access token (PAT) authentication:
# source_connection_config=ConfluenceConnectionConfig(
# access_config=ConfluenceAccessConfig(
# token=os.getenv("CONFLUENCE_PERSONAL_ACCESS_TOKEN")
# ),
# url=os.getenv("CONFLUENCE_URL")
# ),
# For password authentication:
# source_connection_config=ConfluenceConnectionConfig(
# access_config=ConfluenceAccessConfig(
# password=os.getenv("CONFLUENCE_PASSWORD")
# ),
# url=os.getenv("CONFLUENCE_URL"),
# username=os.getenv("CONFLUENCE_USERNAME"),
# cloud=False
# ),
partitioner_config=PartitionerConfig(
partition_by_api=True,
api_key=os.getenv("UNSTRUCTURED_API_KEY"),
partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
additional_partition_args={
"split_pdf_page": True,
"split_pdf_allow_failed": True,
"split_pdf_concurrency_level": 15
}
),
chunker_config=ChunkerConfig(chunking_strategy="by_title"),
embedder_config=EmbedderConfig(embedding_provider="huggingface"),
uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
).run()
```
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:
* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).
Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`
* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).
Unstructured also requires an Unstructured API key and API URL, by adding the following:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.
You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.
The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy[Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).
If you are using a **Business** account, the process
for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).