This page was recently updated. What do you think about it? Let us know!.
Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem. The requirements are as follows.
  • A Snowflake account and its account identifier. To get the identifier for the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click your username, and then click Account > View account details.
    3. On the Account tab, note the value of the Account Identifier field.
    Alternatively, the following Snowflake query returns the current account’s identifier: 
    SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  • A Snowflake user, which can be a service user (recommended) or a human user. To create a service user entry and get their login name (not username):
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Projects > Worksheets.
    3. Click the + button to create a SQL worksheet.
    4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:
      • Replace <service-user-name> with some name for the service user.
      • Replace <default-role-name> with the name of any default role for the service user to use.
      CREATE USER <service-user-name>
DEFAULT_ROLE = "<default-role-name>"
TYPE = SERVICE
    5. Click the arrow icon to run the worksheet, which creates the service user.
    6. To get their login name, on the navigation menu, click Admin > Users & Roles.
    7. On the Users tab, in the list of available users, click the name of the target user.
    8. In the About tile, note the Login Name for the user.
    To create a human user entry and get their login name (not username):
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Users & roles.
    3. Click the Users tab.
    4. Click + User.
    5. Follow the on-screen guidance to specify the user’s settings.
    6. Click Create User.
    7. To get their login name, on the navigation menu, click Admin > Users & Roles.
    8. On the Users tab, in the list of available users, click the name of the target user.
    9. In the About tile, note the Login Name for the user.
  • A programmatic access token (PAT) for the Snowflake user. To create a programmatic access token (PAT) for a user:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Users & Roles.
    3. On the Users tab, in the list of available users, click the name of the target user.
    4. In the Programmatic access tokens tile, click the Generate new token button.
    5. Follow the on-screen guidance to specify the PAT’s settings.
      You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later. Once this PAT expires, the connector will stop working. To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and update your connector’s settings with your new PAT’s value.Unstructured does not notify you when a PAT is about to expire or has already expired. You are responsible for tracking your PATs’ expiration dates and taking corrective action before they expire.
    6. Click Generate.
    7. Copy the generated PAT’s value to a secure location, as you will not be able to access it again. If you lose this PAT’s value, you will need to repeat this procedure to generate a new, replacement one.
    The PAT will not work unless the Snowflake account also has a valid network rule along with a valid network policy attached to that rule. The network rule must also be activated on the Snowflake account to begin taking effect. To create a valid network rule:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Security > Network Rules.
    3. Click + Network Rule.
    4. Enter some name for the network rule.
    5. For Type, select IPv4.
    6. For Mode, select Ingress.
    7. For Identifiers, next to the magnifying glass icon, enter 0.0.0.0/0, and then press Enter.
      The 0.0.0.0/0 value allows all IP addresses to access the Snowflake account. You can specify a more specific IP address range if you prefer. However, this more specific IP address range will apply to all users, including the user for which you created the PAT.
    8. Click Create Network Rule.
    To create a valid network policy, attaching the preceding network rule to this policy at the same time:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Security > Network Policies.
    3. Click + Network Policy.
    4. Enter some name for the network policy.
    5. Make sure Allowed is selected.
    6. In the Select rule drop-down list, select the precedingnetwork rule to attach to this network policy.
    7. Click Create Network Policy.
    To activate the network rule in the account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Security > Network Policies.
    3. Click the name of the precedingnetwork policy to activate.
    4. In the policy’s side panel, click the ellipsis (three dots) icon, and then click Activate On Account.
    5. Click Activate policy.
  • (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake user’s login name (not username) and the user’s password in the account. This user must be a human user. Passwords are not supported for service users.
  • The name of the Snowflake role that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.
    • To create a database in Snowflake, the role needs to be granted CREATE DATABASE privilege at the current account level; and USAGE privilege on the warehouse that is used to create the database.
    • To create a schema in a database in Snowflake, the role needs to be granted USAGE privilege on the database and the warehouse that is used to create the schema; and CREATE SCHEMA on the database.
    • To create a table in a schema in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to create the table; and CREATE TABLE on the schema.
    • To write to a table in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to write to the table; and INSERT on the table.
    • To read from a table in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to write to the table; and SELECT on the table.
    To view a list of available roles in the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Users & Roles.
    3. Click the Roles tab.
    Alternatively, the following Snowflake query returns a list of available roles in the current account: 
    SHOW ROLES;
    Grant privileges to a role. Learn more.
  • The Snowflake warehouse’s hostname and its port number in the account. To view a list of available warehouses in the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Admin > Warehouses. This view does not provide access to the warehouses’ hostnames or port numbers. To get this information, you must run a Snowflake query.
    The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a type of SNOWFLAKE_DEPLOYMENT: 
    SELECT t.VALUE:type::VARCHAR as type,
       t.VALUE:host::VARCHAR as host,
       t.VALUE:port as port
FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  • The name of the Snowflake database in the account. To view a list of available databases in the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Data > Databases.
    Alternatively, the following Snowflake query returns a list of available databases in the current account: 
    SHOW DATABASES;
  • The name of the schema in the database. To view a list of available schemas for a database in the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Data > Databases.
    3. Expand the name of the target database.
    Alternatively, the following Snowflake query returns a list of available schemas in the current account: 
    SHOW SCHEMAS;
    The following Snowflake query returns a list of available schemas for the database named <database_name> in the current account: 
    SHOW SCHEMAS IN DATABASE <database_name>;
  • The name of the table in the schema. To view a list of available tables for a schema in a database in the current Snowflake account:
    1. Log in to Snowsight with your Snowflake account.
    2. In Snowsight, on the navigation menu, click Data > Databases.
    3. Expand the name of the database that contains the target schema.
    4. Expand the name of the target schema.
    5. Expand Tables.
    Alternatively, the following Snowflake query returns a list of available tables for the schema named <schema_name> in the datbase named <database_name> in the current account: 
    SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
    Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The recommended table schema for Unstructured is as follows. In the following CREATE TABLE statement, replace the following placeholders with the appropriate values:
    • <database_name>: The name of the target database in the Snowflake account.
    • <schema_name>: The name of the target schema in the database.
    • <number-of-dimensions>: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
      specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search, this value must match the number of dimensions that you plan to have Snowflake generate or search against.
    SQL
    CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
  ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
  RECORD_ID VARCHAR,
  ELEMENT_ID VARCHAR,
  TEXT VARCHAR,
  EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
  TYPE VARCHAR,
  SYSTEM VARCHAR,
  LAYOUT_WIDTH DECIMAL,
  LAYOUT_HEIGHT DECIMAL,
  POINTS VARCHAR,
  URL VARCHAR,
  VERSION VARCHAR,
  DATE_CREATED TIMESTAMP_TZ,
  DATE_MODIFIED TIMESTAMP_TZ,
  DATE_PROCESSED TIMESTAMP_TZ,
  PERMISSIONS_DATA VARCHAR,
  RECORD_LOCATOR VARCHAR,
  CATEGORY_DEPTH INTEGER,
  PARENT_ID VARCHAR,
  ATTACHED_FILENAME VARCHAR,
  FILETYPE VARCHAR,
  LAST_MODIFIED TIMESTAMP_TZ,
  FILE_DIRECTORY VARCHAR,
  FILENAME VARCHAR,
  LANGUAGES ARRAY,
  PAGE_NUMBER VARCHAR,
  LINKS VARCHAR,
  PAGE_NAME VARCHAR,
  LINK_URLS ARRAY,
  LINK_TEXTS ARRAY,
  SENT_FROM ARRAY,
  SENT_TO ARRAY,
  SUBJECT VARCHAR,
  SECTION VARCHAR,
  HEADER_FOOTER_TYPE VARCHAR,
  EMPHASIZED_TEXT_CONTENTS ARRAY,
  EMPHASIZED_TEXT_TAGS ARRAY,
  TEXT_AS_HTML VARCHAR,
  REGEX_METADATA VARCHAR,
  DETECTION_CLASS_PROB DECIMAL,
  IMAGE_BASE64 VARCHAR,
  IMAGE_MIME_TYPE VARCHAR,
  ORIG_ELEMENTS VARCHAR,
  IS_CONTINUATION BOOLEAN
);
  • The name of the column in the table that uniquely identifies each record (for example, RECORD_ID).
The Snowflake connector dependencies:
CLI, Python
pip install "unstructured-ingest[snowflake]"
You might also need to install additional dependencies, depending on your needs. Learn more. These environment variables:
  • SNOWFLAKE_ACCOUNT - The ID of the target Snowflake account, represented by --account (CLI) or account (Python).
  • SNOWFLAKE_USER - The name of the target Snowflake user, represented by --user (CLI) or user (Python).
  • SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN - The user’s programmatic access token (PAT), represented by --password (CLI) or password (Python).
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  • SNOWFLAKE_ROLE - The target role for the user, represented by --role (CLI) or role (Python).
  • SNOWFLAKE_HOST - The hostname for the target Snowflake warehouse, represented by --host (CLI) or host (Python).
  • SNOWFLAKE_PORT - The warehouse’s port number, represented by --port (CLI) or port (Python). The default is 443 if not otherwise specified.
  • SNOWFLAKE_DATABASE - The name of the target Snowflake database, represented by --database (CLI) or database (Python).
  • SNOWFLAKE_SCHEMA - The name of the target schema in the database, represented by --schema (CLI) or schema (Python).
  • SNOWFLAKE_TABLE - The name of the target table in the schema, represented by --table-name (CLI) or table_name (Python). For the destination connector, the default is elements if not otherwise specified.
  • SNOWFLAKE_RECORD_ID_KEY - The name of the column in the table that uniquely identifies each record, represented by:
    • For the source connector, --id-column (CLI) or id_column (Python).
    • For the destination connector, --record-id-key (CLI) or record_id_key (Python). For the destination connector, the default is record_id if not otherwise specified.
Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector: This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page. 
#!/usr/bin/env bash

unstructured \
  snowflake \
    --account $SNOWFLAKE_ACCOUNT \
    --user $SNOWFLAKE_USER \
    --password $SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN \
    --role $SNOWFLAKE_ROLE \
    --host $SNOWFLAKE_HOST \
    --port $SNOWFLAKE_PORT \
    --database $SNOWFLAKE_DATABASE \
    --schema $SNOWFLAKE_SCHEMA \
    --batch-size 50 \
    --table-name $SNOWFLAKE_TABLE \
    --id-column $SNOWFLAKE_RECORD_ID_KEY \
    --download-dir $LOCAL_FILE_DOWNLOAD_DIR\
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --output-dir $LOCAL_FILE_OUTPUT_DIR
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api option (CLI) or partition_by_api (Python) parameter to specify where files are processed:
  • To do local file processing, omit --partition-by-api (CLI) or partition_by_api (Python), or explicitly specify partition_by_api=False (Python). Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL
  • To send files to the Unstructured Partition Endpoint for processing, specify --partition-by-api (CLI) or partition_by_api=True (Python). Unstructured also requires an Unstructured API key and API URL, by adding the following:
    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL, representing your API key and API URL, respectively.
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to Starter and Team accounts.The default API URL for Unstructured Ingest is https://api.unstructuredapp.io/general/v0/general, which is the API URL for the Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at support@unstructured.io.If you do not have an API key, get one now.If you are using an Enterprise account, the process for generating Unstructured API keys, and the Unstructured API URL that you use, are different. For instructions, see your Unstructured account administrator, or email Unstructured Support at support@unstructured.io.