s3:ListBucket
and s3:GetObject
for that bucket.
For write access, the authenticated AWS IAM user must have at minimum the permission of s3:PutObject
for that bucket. Permissions
can be granted in one of the following ways:
AccessKeyId
), AWS secret access key (SecretAccessKey
), and AWS STS session token (SessionToken
).
--key
, --secret
, and --token
(CLI) or key
, secret
, and token
(Python) in your command or code for the
corresponding S3 source or destination connector.protocol://bucket/
(for example, s3://my-bucket/
).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as protocol://bucket/path/to/folder/
(for example, s3://my-bucket/my-folder/
).
AWS_S3_URL
- The path to the S3 bucket or folder, formatted as s3://my-bucket/
(if the files are in the bucket’s root) or s3://my-bucket/my-folder/
.
AWS_ACCESS_KEY_ID
- The AWS access key ID for the authenticated AWS IAM user, represented by --key
(CLI) or key
(Python).AWS_SECRET_ACCESS_KEY
- The corresponding AWS secret access key, represented by --secret
(CLI) or secret
(Python).AWS_STS_TOKEN
- If required, the AWS STS session token for temporary access, represented by --token
(CLI) or token
(Python).--anonymous
(CLI) or anonymous=True
(Python) instead.
root
to that
specific username.
In this policy, replace the following:
<my-account-id>
with your AWS account ID.<my-bucket-name>
in two places with the name of your bucket.create-s3-bucket.sh
.
To change the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this script, replace the following:
<my-account-id>
with your AWS account ID.<my-unique-bucket-name>
with the name of your bucket.<us-east-1>
with your AWS Region.create-s3-bucket.yaml
. To change
the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
x-amz-meta-
and is
followed by a unique name.
For more information about how to add or replace user-defined metadata for a file in S3, see the following:
Unstructured outputs any user-defined metadata that it finds for a file into the metadata.data_source.record_locator.metadata
field of
the document elements’ output for the corresponding file. For example, if Unstructured processes a file with the user-defined metadata
x-amz-meta-mymetadata
name set to the value myvalue
, Unstructured outputs the following into the metadata.data_source.record_locator.metadata
field of
the document elements’ output for the corresponding file:
--partition-by-api
option (CLI) or partition_by_api
(Python) parameter to specify where files are processed:
--partition-by-api
(CLI) or partition_by_api
(Python), or explicitly specify partition_by_api=False
(Python).
Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
--api-key $UNSTRUCTURED_API_KEY
(CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)UNSTRUCTURED_API_KEY
and UNSTRUCTURED_API_URL
--partition-by-api
(CLI) or partition_by_api=True
(Python).
Unstructured also requires an Unstructured API key and API URL, by adding the following:
--api-key $UNSTRUCTURED_API_KEY
(CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)UNSTRUCTURED_API_KEY
and UNSTRUCTURED_API_URL
, representing your API key and API URL, respectively.https://api.unstructuredapp.io/general/v0/general
, which is the API URL for the Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.If you do not have an API key, get one now.If the Unstructured API is self-hosted, the process
for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
For details, contact Unstructured Sales at
sales@unstructured.io.