Process a subset of files

​

Task

You want to process only files with specified extensions, only files at or below a specified size, or both.

​

Approach

For the Ingest CLI, use the following command options. For the Ingest Python library, use the following parameters for the FiltererConfig object.

• Use --file-glob (CLI) or file_glob (Python) to specify the list of file extensions to process.
• Use --max-file-size (CLI) or max_file_size (Python) to specify the maximum size of files to process, in bytes.

​

To run this example

The following example processes only .pdf and .eml files that have a file size of 100 KB or less. To run this example, you should have a directory with a mixture of files, including at least one .pdf file and one .eml file, and with at least one of these files having a file size of 100 KB or less.

​

Code

unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --file-glob "*.pdf,*.eml" \
    --max-file-size 100000 \
    --partition-by-api \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --api-key $UNSTRUCTURED_API_KEY \
    --strategy hi_res