VAST

​

Requirements

• A VAST user account with permissions to the bucket to which you want to connect.
• A bucket that contains the objects to which you want to connect. A bucket is a storage resource that uses Amazon’s Simple Storage Service (S3) as an access protocol. To create a bucket, see Creating S3 Buckets in the VAST Cluster Administrator’s Guide.
• The path to the files in the bucket. If the target files are in the root of the bucket, the path should be formatted as vast://bucket/ (for example, vast://my-bucket/). If the target files are in a folder, format the path to the target folder as vast://bucket/path/to/folder/ (for example, vast://my-bucket/my-folder/).
• The URL for the VAST region in which the bucket is located (for example, https://vast-cluster.my-company.com).
• An enabled access and secret key pair that grants the authenticated user permission to the VAST bucket. To grant a user access to a bucket, and attach identity policies that define their permissions on that bucket, see Managing S3 User Access from the VAST Web UI or Managing S3 Access from the VAST CLI in the VAST Cluster Administrator’s Guide.
  VAST recommends using identity policies and bucket policies to control S3 bucket access.

  • Identity policies are attached to users and groups, and defines the actions the users or groups can perform against buckets.
  • Bucket policies are attached to a view, and define actions that particular users or groups can perform against the view and files or directories under the view.

  For more information, see Overview of S3 Access Management in the VAST Cluster Administrator’s Guide.Identity and bucket policies support a subset of elements listed in Amazon’s IAM JSON Policy Reference. For information about the required JSON format, supported elements and identity and bucket policy examples, see Identity and Bucket Policy Reference in the VAST Cluster Administrator’s Guide.
  For source connectors, the authenticated user must have bucket permissions for the following S3 actions:
  • s3:ListBucket
  • s3:GetObject
  For destination connectors, the authenticated user must have bucket permissions for the following S3 actions:
  • s3:PutObject

​

Examples

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateDestinationRequest
from unstructured_client.models.shared import CreateDestinationConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.destinations.create_destination(
        request=CreateDestinationRequest(
            create_destination_connector=CreateDestinationConnector(
                name="<name>",
                type="vast",
                config={
                    "remote_url": "<remote-ur>",
                    "endpoint_url": "<endpoint-url>",
                    "recursive": <true|false>,
                    "max-objects": <integer>,
                    "key": "<key>",
                    "secret": "<secret>" 
                }
            )
        )
    )

    print(response.destination_connector_information)
# ...

​

Configuration settings