Snowflake
Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.
The requirements are as follows.
-
A Snowflake account and its account identifier.
To get the identifier for the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click your username, and then click Account > View account details.
- On the Account tab, note the value of the Account Identifier field.
Alternatively, the following Snowflake query returns the current account’s identifier:
-
The Snowflake user’s login name (not its username) and its password in the account.
To view the login name for a user:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Admin > Users & Roles.
- On the Users tab, in the list of available users, click the name of the target user.
- In the About tile, note the Login Name for the user.
Alternatively, the following Snowflake query returns information about the user with the username of
<my-user>
, including theirlogin_name
value representing their login name: -
The Snowflake warehouse’s hostname and its port number in the account.
To view a list of available warehouses in the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Admin > Warehouses. This view does not provide access to the warehouses’ hostnames or port numbers. To get this information, you must run a Snowflake query.
The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a
type
ofSNOWFLAKE_DEPLOYMENT
: -
The name of the Snowflake database in the account.
To view a list of available databases in the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Data > Databases.
Alternatively, the following Snowflake query returns a list of available databases in the current account:
-
The name of the schema in the database.
To view a list of available schemas for a database in the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Data > Databases.
- Expand the name of the target database.
Alternatively, the following Snowflake query returns a list of available schemas in the current account:
The following Snowflake query returns a list of available schemas for the database named
<database-name>
in the current account: -
The name of the table in the schema.
To view a list of available tables for a schema in a database in the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Data > Databases.
- Expand the name of the database that contains the target schema.
- Expand the name of the target schema.
- Expand Tables.
Alternatively, the following Snowflake query returns a list of available tables for the schema named
<schema-name>
in the datbase named<database-name>
in the current account:Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The recommended table schema for Unstructured is as follows:
SQL -
The name of the column in the table that uniquely identifies each record (for example,
RECORD_ID
). -
The name of the Snowflake role that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.
- To create a database in Snowflake, the role needs to be granted
CREATE DATABASE
privilege at the current account level; andUSAGE
privilege on the warehouse that is used to create the database. - To create a schema in a database in Snowflake, the role needs to be granted
USAGE
privilege on the database and the warehouse that is used to create the schema; andCREATE SCHEMA
on the database. - To create a table in a schema in Snowflake, the role needs to be granted
USAGE
privilege on the database and schema and the warehouse that is used to create the table; andCREATE TABLE
on the schema. - To write to a table in Snowflake, the role needs to be granted
USAGE
privilege on the database and schema and the warehouse that is used to write to the table; andINSERT
on the table. - To read from a table in Snowflake, the role needs to be granted
USAGE
privilege on the database and schema and the warehouse that is used to write to the table; andSELECT
on the table.
To view a list of available roles in the current Snowflake account:
- Log in to Snowsight with your Snowflake account.
- In Snowsight, on the navigation menu, click Admin > Users & Roles.
- Click the Roles tab.
Alternatively, the following Snowflake query returns a list of available roles in the current account:
- To create a database in Snowflake, the role needs to be granted
The Snowflake connector dependencies:
You might also need to install additional dependencies, depending on your needs. Learn more.
These environment variables:
-
SNOWFLAKE_ACCOUNT
- The ID of the target Snowflake account, represented by--account
(CLI) oraccount
(Python). -
SNOWFLAKE_USER
- The name of the target Snowflake user, represented by--user
(CLI) oruser
(Python). -
SNOWFLAKE_PASSWORD
- The user’s password, represented by--password
(CLI) orpassword
(Python). -
SNOWFLAKE_ROLE
- The target role for the user, represented by--role
(CLI) orrole
(Python). -
SNOWFLAKE_HOST
- The hostname for the target Snowflake warehouse, represented by--host
(CLI) orhost
(Python). -
SNOWFLAKE_PORT
- The warehouse’s port number, represented by--port
(CLI) orport
(Python). The default is443
if not otherwise specified. -
SNOWFLAKE_DATABASE
- The name of the target Snowflake database, represented by--database
(CLI) ordatabase
(Python). -
SNOWFLAKE_SCHEMA
- The name of the target schema in the database, represented by--schema
(CLI) orschema
(Python). -
SNOWFLAKE_TABLE
- The name of the target table in the schema, represented by--table-name
(CLI) ortable_name
(Python). For the destination connector, the default iselements
if not otherwise specified. -
SNOWFLAKE_RECORD_ID_KEY
- The name of the column in the table that uniquely identifies each record, represented by:- For the source connector,
--id-column
(CLI) orid_column
(Python). - For the destination connector,
--record-id-key
(CLI) orrecord_id_key
(Python). For the destination connector, the default isrecord_id
if not otherwise specified.
- For the source connector,
Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:
This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api
option (CLI) or partition_by_api
(Python) parameter to specify where files are processed:
-
To do local file processing, omit
--partition-by-api
(CLI) orpartition_by_api
(Python), or explicitly specifypartition_by_api=False
(Python).Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
--api-key $UNSTRUCTURED_API_KEY
(CLI) orapi_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) orpartition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)- The environment variables
UNSTRUCTURED_API_KEY
andUNSTRUCTURED_API_URL
-
To send files to Unstructured API services for processing, specify
--partition-by-api
(CLI) orpartition_by_api=True
(Python).Unstructured API services also requires an Unstructured API key and API URL, by adding the following:
--api-key $UNSTRUCTURED_API_KEY
(CLI) orapi_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) orpartition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)- The environment variables
UNSTRUCTURED_API_KEY
andUNSTRUCTURED_API_URL
, representing your API key and API URL, respectively.
Was this page helpful?