Kafka

Send processed data from Unstructured to Kafka.

The requirements are as follows.

• A Kafka cluster in Confluent Cloud. (Create a cluster.)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

• The hostname and port number of the bootstrap Kafka cluster to connect to..

• The name of the topic to read messages from or write messages to on the cluster. Create a topic. Access available topics.

• For authentication, an API key and secret.

To create a Kafka destination connector, see the following examples.

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateDestinationRequest
from unstructured_client.models.shared import (
    CreateDestinationConnector,
    DestinationConnectorType,
    KafkaCloudDestinationConnectorConfigInput
)

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.destinations.create_destination(
        request=CreateDestinationRequest(
            create_destination_connector=CreateDestinationConnector(
                name="<name>",
                type=DestinationConnectorType.KAFKA_CLOUD,
                config=KafkaCloudDestinationConnectorConfigInput(
                    bootstrap_servers="<bootstrap-server>",
                    port=<port>,
                    group_id="<group-id>",
                    kafka_api_key="<kafka-api-key>",
                    secret="<secret>",
                    topic="<topic>",
                    batch_size=<batch-size>
                )
            )
        )
    )

    print(response.destination_connector_information)

Replace the preceding placeholders as follows:

• <name> (required) - A unique name for this connector.
• <bootstrap-server> - The hostname of the bootstrap Kafka cluster to connect to.
• <port> - The port number of the bootstrap Kafka cluster to connect to. The default is 9092 if not otherwise specified.
• <group-id> - The ID of the consumer group. A consumer group is a way to allow a pool of consumers to divide the consumption of data over topics and partitions. The default is default_group_id if not otherwise specified.
• <kafka-api-key> - For authentication, the API key for access to the cluster.
• <secret> - For authentication, the secret for access to the cluster.
• <topic> - The name of the topic to read messages from or write messages to on the cluster.
• <batch-size> (destination connector only) - The maximum number of messages to send in a single batch. The default is 100 if not otherwise specified.
• <num-messages-to-consume> (source connector only) - The maximum number of messages that the consumer will try to consume. The default is 100 if not otherwise specified.