If you’re new to Unstructured, read this note first.
Before you can create a destination connector, you must first sign in to your Unstructured account:
After you sign in, the Unstructured user interface (UI) appears, which you use to create your destination connector.
After you create the destination connector, add it along with a source connector to a workflow. Then run the worklow as a job. To learn how, try out the hands-on UI quickstart or watch the 4-minute video tutorial.
You can also create destination connectors with the Unstructured API. Learn how.
If you need help, reach out to the community on Slack, or contact us directly.
You are now ready to start creating a destination connector! Keep reading to learn how.
Send processed data from Unstructured to Amazon S3.
The requirements are as follows.
The following video shows how to fulfill the minimum set of Amazon S3 requirements:
The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but not recommended); or generate an AWS STS session token for temporary access, if required by your organization’s security requirements. For more information about requirements, see the following:
An AWS account. Create an AWS account.
An S3 bucket. Create an S3 bucket. Additional approaches are in the following video and in the how-to sections at the end of this page.
Anonymous (supported but not recommended) or authenticated access to the bucket.
For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of s3:ListBucket
and s3:GetObject
for that bucket. Learn how.
For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
minimum the permission of s3:PutObject
for that bucket. Learn how.
For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account. Create an AWS access key and secret access key.
For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. Create a session token.
AWS STS credentials can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid will no longer work with the corresponding S3 connector. You must get a new set of credentials to replace the expired ones by calling GetSessionToken in the AWS STS API. To overwrite the expired credentials with the new set:
--key
, --secret
, and --token
(CLI) or key
, secret
, and token
(Python) in your command or code for the
corresponding S3 source or destination connector.If the target files are in the root of the bucket, the path to the bucket, formatted as protocol://bucket/
(for example, s3://my-bucket/
).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as protocol://bucket/path/to/folder/
(for example, s3://my-bucket/my-folder/
).
If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has authenticated access to the folder as well. Enable authenticated folder access.
To create the destination connector:
Fill in the following fields:
s3://my-bucket/
(if the files are in the bucket’s root) or s3://my-bucket/my-folder/
.To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to an existing S3 bucket, do the following.
Sign in to the AWS Management Console.
Open the Amazon S3 Console.
Browse to the existing bucket and open it.
Click the Permissions tab.
In the Bucket policy area, click Edit.
In the Policy text area, copy the following JSON-formatted policy.
To change the following policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this policy, replace the following:
<my-account-id>
with your AWS account ID.<my-bucket-name>
in two places with the name of your bucket.Click Save changes.
To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to the bucket, do the following.
Save the following YAML to a file on your local machine, for example create-s3-bucket.yaml
. To change
the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
Sign in to the AWS Management Console.
Open the AWS CloudFormation Console.
Click Create stack > With new resources (standard).
On the Create stack page, with Choose an existing template already selected, select Upload a template file.
Click Choose file, and browse to and select the YAML file from your local machine.
Click Next.
Enter a unique Stack name and BucketName.
Click Next two times.
Click Submit.
Wait until the Status changes to CREATE_COMPLETE.
After the bucket is created, you can delete the YAML file, if you want.
To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to the bucket, do the following.
Copy the following script to a file on your local machine, for example a file named create-s3-bucket.sh
.
To change the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this script, replace the following:
<my-account-id>
with your AWS account ID.<my-unique-bucket-name>
with the name of your bucket.<us-east-1>
with your AWS Region.Run the script, for example:
After the bucket is created, you can delete the script file, if you want.
If you’re new to Unstructured, read this note first.
Before you can create a destination connector, you must first sign in to your Unstructured account:
After you sign in, the Unstructured user interface (UI) appears, which you use to create your destination connector.
After you create the destination connector, add it along with a source connector to a workflow. Then run the worklow as a job. To learn how, try out the hands-on UI quickstart or watch the 4-minute video tutorial.
You can also create destination connectors with the Unstructured API. Learn how.
If you need help, reach out to the community on Slack, or contact us directly.
You are now ready to start creating a destination connector! Keep reading to learn how.
Send processed data from Unstructured to Amazon S3.
The requirements are as follows.
The following video shows how to fulfill the minimum set of Amazon S3 requirements:
The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but not recommended); or generate an AWS STS session token for temporary access, if required by your organization’s security requirements. For more information about requirements, see the following:
An AWS account. Create an AWS account.
An S3 bucket. Create an S3 bucket. Additional approaches are in the following video and in the how-to sections at the end of this page.
Anonymous (supported but not recommended) or authenticated access to the bucket.
For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of s3:ListBucket
and s3:GetObject
for that bucket. Learn how.
For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
minimum the permission of s3:PutObject
for that bucket. Learn how.
For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account. Create an AWS access key and secret access key.
For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. Create a session token.
AWS STS credentials can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid will no longer work with the corresponding S3 connector. You must get a new set of credentials to replace the expired ones by calling GetSessionToken in the AWS STS API. To overwrite the expired credentials with the new set:
--key
, --secret
, and --token
(CLI) or key
, secret
, and token
(Python) in your command or code for the
corresponding S3 source or destination connector.If the target files are in the root of the bucket, the path to the bucket, formatted as protocol://bucket/
(for example, s3://my-bucket/
).
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as protocol://bucket/path/to/folder/
(for example, s3://my-bucket/my-folder/
).
If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has authenticated access to the folder as well. Enable authenticated folder access.
To create the destination connector:
Fill in the following fields:
s3://my-bucket/
(if the files are in the bucket’s root) or s3://my-bucket/my-folder/
.To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to an existing S3 bucket, do the following.
Sign in to the AWS Management Console.
Open the Amazon S3 Console.
Browse to the existing bucket and open it.
Click the Permissions tab.
In the Bucket policy area, click Edit.
In the Policy text area, copy the following JSON-formatted policy.
To change the following policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this policy, replace the following:
<my-account-id>
with your AWS account ID.<my-bucket-name>
in two places with the name of your bucket.Click Save changes.
To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to the bucket, do the following.
Save the following YAML to a file on your local machine, for example create-s3-bucket.yaml
. To change
the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
Sign in to the AWS Management Console.
Open the AWS CloudFormation Console.
Click Create stack > With new resources (standard).
On the Create stack page, with Choose an existing template already selected, select Upload a template file.
Click Choose file, and browse to and select the YAML file from your local machine.
Click Next.
Enter a unique Stack name and BucketName.
Click Next two times.
Click Submit.
Wait until the Status changes to CREATE_COMPLETE.
After the bucket is created, you can delete the YAML file, if you want.
To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the corresponding AWS account to read and write to the bucket, do the following.
Copy the following script to a file on your local machine, for example a file named create-s3-bucket.sh
.
To change the following bucket policy to restrict it to a specific user in the AWS account, change root
to that
specific username.
In this script, replace the following:
<my-account-id>
with your AWS account ID.<my-unique-bucket-name>
with the name of your bucket.<us-east-1>
with your AWS Region.Run the script, for example:
After the bucket is created, you can delete the script file, if you want.