SQL

Batch process all your records to store structured outputs in a PostgreSQL or SQLite schema.

Insert query is currently limited to append.

You will need:

The PostgreSQL or SQLite prerequisites.

For PostgreSQL:

• A PostgreSQL instance. Install PostgreSQL.
• The host name and port number for the instance. These values are in the postgresql.conf file’s listen_addresses and port settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named PGHOST and PGPORT on the same machine as the instance.
• A database in the instance. Create a database.
• A table in the database. Create a table. See also the Sample index schema.
• A user in the database, and a password for the user. Create a user.
• Database access for the user. Give database access to a user.

For SQLite:

• A SQLite instance. Download and install SQLite.
• A SQLite database. Create a database.
• The path to the database’s .db file.
• A table in the database. Create a table. See also the Sample index schema.

The PostgreSQL and SQLite connector dependencies:

pip install "unstructured-ingest[postgres]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

For PostgreSQL:

• SQL_DB_TYPE - The SQL database type. For PostgreSQL, this is always postgresql, represented by --db-type (CLI) or db_type (Python).
• PGHOST - The host name, represented by --host (CLI) or host (Python).
• PGPORT - The port number, represented by --port (CLI) or port (Python).
• PGUSER - The username, represented by --username (CLI) or username (Python).
• PGPASSWORD - The user’s password, represented by --password (CLI) or password (Python).
• PGDATABASE - The name of the database, represented by --database (CLI) or database (Python).

For SQLite:

• SQL_DB_TYPE - The SQL database type. For SQLite, this is always sqlite, represented by --db-type (CLI) or db_type (Python).
• SQLITE_DB_PATH - The path to the database’s .db file, represented by --database (CLI) or database (Python).

These environment variables:

• UNSTRUCTURED_API_KEY - Your Unstructured API key value.
• UNSTRUCTURED_API_URL - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

​

Sample index schema

To make sure the schema of the index matches the data being written to it, a sample schema can be used.