OpenSearch

Documentation Index

Fetch the complete documentation index at: https://docs.unstructured.io/llms.txt

Use this file to discover all available pages before exploring further.

​

Requirements

​

Set up OpenSearch

• Unstructured UI and Unstructured API: Non-local OpenSearch only. Local OpenSearch instances are not supported.
• Unstructured Ingest: Both local and non-local OpenSearch instances are supported.

​

Set up an AWS OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand Managed clusters, and then click Dashboard.
4. Click Create domain.
5. In the Name tile, for Domain name, enter some unique domain name for your new OpenSearch domain.
6. In the Domain creation method tile, select Easy create. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With Standard create, you must enable FGAC manually.) Standard create method documentation.
7. In the Engine options tile, for Version, AWS recommends that you select the latest version.
8. In the Network tile, for Network, select a network access method. For faster setup, this example uses the Public access method. VPC access method documentation.
9. For IP address type, select Dual-stack mode.
10. In the Fine-grained access control (FGAC) tile, do one of the following:
    • If you want to use an existing AWS IAM user in the AWS account as the domain’s master user, then for Master user, select Set IAM ARN as master user. Then enter the IAM ARN for the master user in the IAM ARN box.
    • If you want to create a master user and password as the domain’s master user instead, then for Master user, select Create master user. Then specify some username and password for this new master user by filling in the Master username, Master password, and Confirm master password fields. Make sure to save the master user’s password in a secure location.
11. Click Create.
12. After the domain is created, you must allow Unstructured to access the domain, as follows: a. If the new domain’s settings page is not already showing, open it as follows: in your Amazon OpenSearch Service console, on the sidebar, expand Managed clusters, and then click Domains. Then, in the list of available domains, click the name of the newly created domain.
    b. On the Security configuration tab, click Edit.
    c. In the Access policy tile, for Domain access policy, select Only use fine-grained access control.
    d. Click Clear policy. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).
    e. Click Save changes.

​

Set up an Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand Serverless, and then click Dashboard.
4. Click Create collection.
5. In the Collection details tile, for Collection name, enter some unique name for your new OpenSearch Serverless collection. Optionally, for Description, enter some meaningful description for your new collection.
6. For Collection type, select Search.
   Unstructured does not support the Vector search collection type. If you need vector search support, you can either continue with these steps to use the Search collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead. However, note that the Amazon OpenSearch Serverless Search collection type is not as optimal as the Vector search collection type.
7. In the Collection creation method tile, select Standard create.
8. For Encryption, choose an AWS KMS key type.
9. For Network access settings, choose an Access type.
10. For Resource type, select both Enable access to OpenSearch endpoint and Enable access to OpenSearch Dashboards.
11. Click Next.
12. In the Definition method tile, select JSON.
13. In the JSON editor box, enter the following JSON, replacing the following placeholders:
    • Replace <collection-name> with the name of the new OpenSearch Serverless collection.
    • Replace <account-id> with the target AWS account ID.
    • Replace <user-id> with the ID of the target AWS IAM user.
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    
14. Click Next.
15. For Data access policy settings, select Create as a new data access policy.
16. In the Name and description tile, enter some unique name and an optional description for the new data access policy.
17. Click Next.
18. Enter any desired index details, and click Next again. For example: a. For Index name, enter the name of the new index in the collection.
    b. For Automatic Semantic Enrichment fields, click Add, enter embeddings for Automatic Semantic Enrichment field name, click Add, and click Confirm.
    c. For Lexical search fields, click Add, enter text for Field name and select Text for Data type, click Add, and click Confirm.
    
19. Click Submit.

​

Set up a local OpenSearch instance

​

Get the host URL

​

Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand Managed clusters, and then click Dashboard.
4. In the list of available domains, click the name of your domain.
5. In the General information tile, copy the value of Domain endpoint v2 (dual stack).

​

Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand Serverless, and then click Dashboard.
4. In the list of available collections, click the name of your collection.
5. On the Overview tab, in the Endpoint tile, copy the value of OpenSearch endpoint.

​

Local OpenSearch instance

​

Create a search index

• Replace <host> with the instance’s host URL.
• Replace <port> with the instance’s port number, which is typically 443 (for encrypted connections, and less commonly 9200 for unencrypted connections).
• Replace <master-username> with the master user’s name, and replace <master-password> with the master user’s password.
• Replace <index-name> with the name of the new search index on the instance.
• Replace <index-schema> with the schema for the new search index on the instance. A schema is optional; see the explanation following this curl command for more information.

curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']

"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}

• element_id
• record_id, which is required by Unstructured for intelligent record updates.
• type, which is not required, but highly recommended.
• text
• embeddings if embeddings are generated; make sure to set dimension to the same number of dimensions as the embedding model generates.

{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

• Create an index
• Mappings and field types
• Explicit mapping
• Dynamic mapping
• Unstructured document elements and metadata

​

Set up master user authentication

• The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
• The path to the combined private key and certificate file, or
• The paths to the separate private key and certificate files.

​

Examples

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type="opensearch",
                config={
                    "hosts": ["https://<host>:<port>"],
                    "index_name": "<index-name>",
                    "username": "<username>",
                    "password": "<password>",
                    "aws_access_key_id": "<aws-access-key-id>",
                    "aws_secret_access_key": "<aws-secret-access-key>",
                    "aws_session_token": "<aws-session-token>",
                    "fields": ["<field-name>", "<field-name>"],
                    "use_ssl": <True|False>
                }
            )
        )
    )

    print(response.source_connector_information)

​

Configuration settings

• <name> (required) - A unique name for this connector.
• https://<host>:<port> (required) - The OpenSearch instance’s host URL, which typically takes the form of https://<host>:<port>.
• <index-name> (required) - The name of the search index on the instance.
• <username> - If you’re using basic authentication to the instance, the domain’s master user’s name.
• <password> - If you’re using basic authentication to the instance, the domain’s master user’s password.
• <aws-access-key-id> - If you’re using an existing AWS IAM user as the domain’s master user, the AWS access key ID for the AWS IAM user. If you’re also using AWS STS for authentication, this will be a temporary AWS access key ID.
• <aws-secret-access-key> - If you’re using an existing AWS IAM user as the domain’s master user, the AWS secret access key for the AWS IAM user. If you’re also using AWS STS for authentication, this will be a temporary AWS secret access key.
• <aws-session-token> - If you’re using AWS STS for authentication, the temporary AWS STS session token.
  If you are using the IAM + Session Token authentication method: AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector. You must get a new set of AWS STS credentials to replace the expired ones by, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see Request temporary security credentials.After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector’s settings with the new, refreshed AWS STS credentials.
• <field-name> (source connectors only) - Any specific fields to be accessed in the index.
• <use-ssl> (required) - True if the OpenSearch instance requires an SSL connection; otherwise, false.

​

Set up Enterprise Connect authentication

​

Create the IAM role

1. Choose an External ID — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of +=,.@:\/-; no spaces; 2–1224 characters.
2. In your AWS account, create an IAM role that Unstructured will assume to access your OpenSearch resources (for example, unstructured-connector-role), or use an existing one. For more information, see Create a role using custom trust policies in the AWS IAM User Guide. Attach the following trust policy, replacing the placeholder values:
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   

   Placeholder	Value
   <unstructured-service-role-arn>	The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team.
   <external-id>	The unique value you chose as your External ID.

   The sts:ExternalId condition prevents the confused deputy problem. It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

​

Attach a permissions policy

​

Amazon OpenSearch Service (managed)

• The domain’s access policy is permissive (for example, "Principal": "*")
• Fine-grained access control (FGAC) is the primary access control

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
    }
  ]
}

​

Amazon OpenSearch Serverless

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:<region>:<account-id>:collection/<collection-id>"
    }
  ]
}

​

Grant access to OpenSearch resources

​

Amazon OpenSearch Service (managed) — Map the role inside OpenSearch

• If you selected Easy create in step 6 of Set up an AWS OpenSearch Service domain, FGAC was enabled automatically and these steps are required.
• If your domain has FGAC disabled, skip this section. Your IAM role setup is complete. Proceed to the next section to create your connector.

• For a quick proof-of-concept, use the built-in all_access OpenSearch role, already present in OpenSearch Dashboards under Security → Roles.
• For production, first create a least-privilege OpenSearch role in OpenSearch Dashboards under Security → Roles, scoped to the specific indices your connector will access.

security_exception ... no permissions for [indices:admin/aliases/get] and User [name=arn:aws:iam::...:role/...]

• OpenSearch Dashboards: Go to Security → Roles → select the OpenSearch role → Mapped users tab → add the IAM role ARN under Backend roles. For more information, see Fine-grained access control in Amazon OpenSearch Service in the AWS OpenSearch Service Developer Guide.
• Using the API:
  curl -XPUT "$DOMAIN/_plugins/_security/api/rolesmapping/<opensearch-role-name>" \
    -H "Content-Type: application/json" \
    -u "$ADMIN_USER:$ADMIN_PASS" \
    -d '{"backend_roles":["<assumed-role-arn>"]}'
  
  Replace <opensearch-role-name> with the name of the OpenSearch role that you selected or created in this section. Replace <assumed-role-arn> with the ARN of the IAM role that you created in step 2 of Create the IAM role.
  PUT replaces the entire role mapping. To preserve existing backend roles, first GET the current mapping and include all existing entries in your PUT request.

​

Amazon OpenSearch Serverless — Add the role to a data access policy

# 1. Find the data access policy for your collection
aws opensearchserverless list-access-policies \
  --type data \
  --region <region> \
  --resource "collection/<collection-name>"

# 2. Get the current policy and note the policyVersion value in the response
aws opensearchserverless get-access-policy \
  --type data \
  --name <policy-name> \
  --region <region>

# 3. Update the policy, adding the IAM role ARN to the Principal array
aws opensearchserverless update-access-policy \
  --type data \
  --name <policy-name> \
  --policy-version <version-from-step-2> \
  --region <region> \
  --policy file://policy.json

[
  {
    "Rules": [
      {
        "Resource": ["collection/<collection-name>"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/<collection-name>/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument",
          "aoss:UpdateIndex"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": ["arn:aws:iam::<account-id>:role/<role-name>"],
    "Description": "Unstructured connector access"
  }
]

​

Create the source connector with Enterprise Connect

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type="opensearch",
                config={
                    "hosts": ["https://<host>:<port>"],
                    "index_name": "<index-name>",
                    "role_arn": "<role-arn>",
                    "external_id": "<external-id>",
                    "use_ssl": <True|False>
                }
            )
        )
    )

    print(response.source_connector_information)