Connect Reddit to your preprocessing pipeline, and batch process all your documents using unstructured-ingest to store structured outputs locally on your filesystem.

First, install the Reddit dependencies as shown here.

pip install "unstructured-ingest[reddit]"

You must provide:

  • A client ID and a client secret to authenticate yourself. Learn how to get them here.
  • user-agent: user agent request header to use when calling Reddit API
  • subreddit-name: The name of a subreddit, without the “r\”, e.g. “machinelearning”

Optionally you can choose to specify:

  • search-query: If set, return posts using this query. Otherwise, use hot posts.
  • num-posts: If set, limits the number of posts to pull in.

Make sure to set the --partition-by-api flag and pass in your API key with --api-key:

Additionally, if you’re using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API deployed on Azure or AWS, you also need to specify the API URL via the --partition-endpoint argument.

Was this page helpful?

PostgreSQLS3