s3:ListBucket
and s3:GetObject
for that bucket.
For write access, the authenticated AWS IAM user must have at minimum the permission of s3:PutObject
for that bucket. Permissions
can be granted in one of the following ways:
AccessKeyId
), AWS secret access key (SecretAccessKey
), and AWS STS session token (SessionToken
).
--key
, --secret
, and --token
(CLI) or key
, secret
, and token
(Python) in your command or code for the
corresponding S3 source or destination connector.protocol://bucket/
(for example, s3://my-bucket/
).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as protocol://bucket/path/to/folder/
(for example, s3://my-bucket/my-folder/
).
root
to that
specific username.
In this policy, replace the following:
<my-account-id>
with your AWS account ID.<my-bucket-name>
in two places with the name of your bucket.create-s3-bucket.yaml
. To change
the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
create-s3-bucket.sh
.
To change the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this script, replace the following:
<my-account-id>
with your AWS account ID.<my-unique-bucket-name>
with the name of your bucket.<us-east-1>
with your AWS Region.x-amz-meta-
and is
followed by a unique name.
For more information about how to add or replace user-defined metadata for a file in S3, see the following:
Unstructured outputs any user-defined metadata that it finds for a file into the metadata.data_source.record_locator.metadata
field of
the document elements’ output for the corresponding file. For example, if Unstructured processes a file with the user-defined metadata
x-amz-meta-mymetadata
name set to the value myvalue
, Unstructured outputs the following into the metadata.data_source.record_locator.metadata
field of
the document elements’ output for the corresponding file:
fips
-based
S3 endpoints and, for authenticated access to
S3 FIPS buckets, S3 ambient credentials.
To use the Unstructured (UI) to set up an S3 source or destination connector
to use an S3 FIPS bucket and S3 ambient credentials, do the following:
ALLOW_AMBIENT_CREDENTIALS_S3
, and set its value to true
.https://<bucket-name>.<endpoint>
, for example
https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com
.
If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
formatted as https://<bucket-name>.<endpoint>/path/to/folder
(for example,
https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/
).ALLOW_AMBIENT_CREDENTIALS_S3
, and set its value to true
.
config
parameter’s remote_url
field, specify the path to the S3 FIPS bucket, formatted as
https://<bucket-name>.<endpoint>
, for example
https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com
.
If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
formatted as https://<bucket-name>.<endpoint>/path/to/folder
(for example,
https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/
).
config
parameter, add an ambient_credentials
field, and set its value to true. For example:
<name>
(required) - A unique name for this connector.
<key>
- The AWS access key ID for the authenticated AWS IAM user (required).<secret>
- The AWS secret access key corresponding to the preceding AWS access key ID (required).<token>
- The AWS STS session token for temporary access (required).<endpoint-url>
- A custom URL, if connecting to a non-AWS S3 bucket.
<remote-url>
(required) - The S3 URI to the bucket or folder, formatted as s3://my-bucket/
(if the files are in the bucket’s root) or s3://my-bucket/my-folder/
.
recursive
(source connector only), set to true
to access subfolders within the bucket. The default is false
if not otherwise specified.