This page was recently updated. What do you think about it? Let us know!.

Batch process all your records to store structured outputs in a Neo4j account.

The requirements are as follows.

  • A Neo4j deployment.

    The following video shows how to set up a Neo4j Aura deployment:

  • The username and password for the user who has access to the Neo4j deployment. The default user is typically neo4j.

  • The connection URI for the Neo4j deployment, which starts with neo4j://, neo4j+s://, bolt://, or bolt+s://; followed by localhost or the host name; and sometimes ending with a colon and the port number (such as :7687). For example:

    • For a Neo4j Aura deployment, browse to the target Neo4j instance in the Neo4j Aura account and click Connect > Drivers to get the connection URI, which follows the format neo4j+s://<host-name>. A port number is not used or needed.
    • For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, see Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP for details about how to get the connection URI.
    • For a local Neo4j deployment, the URI is typically bolt://localhost:7687
    • For other Neo4j deployment types, see the deployment provider’s documentation.

    Learn more.

  • The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains two standard databases: one named neo4j for user data and another named system for system data and metadata. Some Neo4j deployment types support more than these two databases per deployment; Neo4j Aura instances do not.

The Neo4j connector dependencies:

CLI, Python
pip install "unstructured-ingest[neo4j]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

  • NEO4J_USERNAME - The name of the target user with access to the target Neo4j deployment, represented by --username (CLI) or username (Python).
  • NEO4J_PASSWORD - The user’s password, represented by --password (CLI) or password (Python).
  • NEO4J_URI - The connection URI for the deployment, represented by --uri (CLI) or uri (Python).
  • NEO4J_DATABASE - The name of the database in the deployment, represented by --database (CLI) or database (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

#!/usr/bin/env bash

# Chunking and embedding are optional.

unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  neo4j \
    --username $NEO4J_USERNAME \
    --password $NEO4J_PASSWORD \
    --uri $NEO4J_URI \ # <scheme>://<host>:<port>
    --database $NEO4J_DATABASE \
    --batch-size 100

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api option (CLI) or partition_by_api (Python) parameter to specify where files are processed:

  • To do local file processing, omit --partition-by-api (CLI) or partition_by_api (Python), or explicitly specify partition_by_api=False (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL

  • To send files to the Unstructured Partition Endpoint for processing, specify --partition-by-api (CLI) or partition_by_api=True (Python).

    Unstructured also requires an Unstructured API key and API URL, by adding the following:

    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL, representing your API key and API URL, respectively.

    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, for example, if you are using a self-hosted instance of the Unstructured API.

    The default API URL for Unstructured Ingest is https://api.unstructuredapp.io/general/v0/general, which is the API URL for the Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.

    If you do not have an API key, get one now.

    If the Unstructured API is self-hosted, the process for generating Unstructured API keys, and the Unstructured API URL that you use, are different. For details, contact Unstructured Sales at sales@unstructured.io.

​
Graph Output

The graph ouput of the Neo4j destination connector is represented in the following diagram:

View the preceding diagram in full-screen mode.

In the preceding diagram:

  • The Document node represents the source file.
  • The UnstructuredElement nodes represent the source file’s Unstructured Element objects, before chunking.
  • The Chunk nodes represent the source file’s Unstructured Element objects, after chunking.
  • Each UnstructuredElement node has a PART_OF_DOCUMENT relationship with the Document node.
  • Each Chunk node also has a PART_OF_DOCUMENT relationship with the Document node.
  • Each UnstructuredElement node has a PART_OF_CHUNK relationship with a Chunk element.
  • Each Chunk node, except for the “last” Chunk node, has a NEXT_CHUNK relationship with its “next” Chunk node.

Learn more about document elements and chunking.

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

MATCH path=(source)-[relationship]->(target)
RETURN path

Query for Chunk to Document relationships:

MATCH (chunk:Chunk)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN chunk, relationship, doc

Query for UnstructuredElement to Document relationships:

MATCH (element:UnstructuredElement)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, relationship, doc

Query for UnstructuredElement to Chunk relationships:

MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
RETURN element, relationship, chunk

Query for Chunk to Chunk relationships:

MATCH (this:Chunk)-[relationship:NEXT_CHUNK]->(previous:Chunk)
RETURN this, relationship, previous

Query for UnstructuredElement to Chunk to Document relationships:

MATCH (element:UnstructuredElement)-[ecrelationship:PART_OF_CHUNK]-(chunk:Chunk)-[cdrelationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, ecrelationship, chunk, cdrelationship, doc

Query for UnstructuredElements containing the text jury, and show their Chunk relationships:

MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE element.text =~ '(?i).*jury.*'
RETURN element, relationship, chunk

Query for the Chunk with the specified id, and show its UnstructuredElement relationships:

MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE chunk.id = '731508bf53637ce4431fe93f6028ebdf'
RETURN element, relationship, chunk

Additionally, for the Unstructured UI and Unstructured Workflow Endpoint, when a Named entity recognition (NER) DAG node is added to a custom workflow, any recognized entities are output as Entity nodes in the graph.

This additional graph ouput of the Neo4j destination connector is represented in the following diagram:

In the preceding diagram:

  • The Chunk node represents one of the source file’s Unstructured Element objects, after chunking.
  • The Entity node represents a recognized entity.
  • A Chunk node can have HAS_ENTITY relationships with Entity nodes.
  • An Entity node can have ENTITY_TYPE relationships with other Entity nodes.

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

MATCH path=(source)-[relationship]->(target)
RETURN path

Query for Entity to Entity relationships:

MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
RETURN child, relationship, parent

Query for Entity nodes containing the text PERSON, and show their Entity relationships:

MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id = 'PERSON'
RETURN child, relationship, parent

Query for Entity nodes containing the text amendment, and show their Chunk relationships:

MATCH (element:Chunk)-[relationship:HAS_ENTITY]->(entity:Entity)
WHERE entity.id =~ '(?i).*amendment.*'
RETURN element, relationship, entity

QUERY FOR Entity nodes containing the text PERSON, and show their Entity to Entity to Chunk relationships:

MATCH (chunk:Chunk)-[ccrelationship:HAS_ENTITY]-(child:Entity)-[cprelationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id =~ 'PERSON'
RETURN chunk, ccrelationship, child, cprelationship, parent