Image descriptions

After partitioning, you can have Unstructured generate text-based summaries of detected images. This summarization is done by using models offered through various model providers. Here is an example of the output of a detected image using GPT-4o. Note specifically the text field that is added. In this text field, type indicates the kind of image that was detected (in this case, a diagram), and description is a summary of the image. Line breaks have been inserted here for readability. The output will not contain these line breaks.

{
  "type": "Image",
  "element_id": "dd1fb72db7937725c9a781906098e6f8",
  "text": "{\n    
    \"type\": \"diagram\",\n    
    \"description\": \"User uploads a flowchart image via a Web Browser, which is then 
      converted to a Base64 Encoded Image. This image is sent to the Back-end System 
      (Node.js) where it is processed by the AI Model Adapter. The output undergoes 
      Validation and Rendering, resulting in Normalized Mermaid Code. AI Assisted 
      Editing is available through an AI Assistant, which allows for the Regenerated 
      Flowchart Image to be viewed again in the Web Browser.\\n\\n
      Text in the image:\\n
        - User\\n
        - Upload flowchart image\\n
        - Web Browser\\n
        - Base64 Encoded Image\\n
        - Back-end System (Node.js)\\n
        - AI Model Adapter\\n
        - Validation and Rendering\\n
        - Normalized Mermaid Code\\n
        - AI Assisted Editing\\n
        - AI Assistant\\n
        - Regenerated Flowchart Image\"\n
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_base64": "/9j...<full results omitted for brevity>...Q==",
    "image_mime_type": "image/jpeg",
    "filename": "7f239e1d4ef3556cc867a4bd321bbc41.pdf",
    "data_source": {}
  }
}

For technical drawings, the text field will contain a type of technical drawing; description with texts containing text strings found in the drawing, tables containing HTML representations of tables found in the drawing, and a description containing a summary of the drawing. Here is an example. Line breaks have been inserted here for readability. The output will not contain these line breaks.

{
  "type": "Image",
  "element_id": "7877acdd762f2afc65b193fa89d8ef46",
  "text": "{\n  
    \"type\": \"technical drawing\",\n  
    \"description\": {\n    
      \"texts\": [\n
        \"RTD 1\",\n      
        \"RTD 2\",\n      
        \"01\",\n      
        \"18.50\\\" Cable Length\",\n      
        \"02\",\n      
        \"1/4\\\" Heat Shrink\",\n      
        \"6X Strip wires 0.100\\\" - 0.115\\\" before crimping\",\n      
        \"2X 1.50\",\n      
        \"22.25\\\" Cable Length\"\n    
      ],\n    
      \"tables\": "<table>
        <thead>
          <tr>
            <th>Item</th>
            <th>Quantity</th>
            <th>Part Number</th>
            <th>Description</th>
            <th>Supplier</th>
            <th>Supplier PN</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td>1</td>
            <td>6</td>
            <td>002622</td>
            <td>Conn Socket 20-24AWG Gold</td>
            <td>Digikey</td>
            <td>WM7082CT-ND</td>
          </tr>
          <tr>
            <td>2</td>
            <td>1</td>
            <td>002647</td>
            <td>Conn Recept 16pos 3mm Dual Row</td>
            <td>Digikey</td>
            <td>WM2490-ND</td>
          </tr>
          <tr>
              <td>3</td>
              <td>2</td>
              <td>102961-01</td>
              <td>M12 Q/D Cable, Elbow, 4-Pole, 5m</td>
              <td>Automation Direct</td>
              <td>EVT222</td>
          </tr>
        </tbody>
      </table>",\n    
      \"description\": \"The technical drawing depicts a wiring setup involving two 
          RTDs (Resistance Temperature Detectors) labeled RTD 1 and RTD 2. Each RTD 
          is connected via cables with specified lengths: RTD 1 has an 18.50-inch 
          cable length, and RTD 2 has a 22.25-inch cable length. The drawing 
          includes annotations for stripping wires, indicating that six wires should 
          be stripped to a length between 0.100 inches and 0.115 inches before 
          crimping. There is a section labeled '1/4\\\" Heat Shrink' and a dimension 
          marked '2X 1.50'. The drawing uses numbered circles to reference specific 
          parts or steps in the process.\"\n  
      }\n
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_base64": "/9j...<full results omitted for brevity>...Q==",
    "image_mime_type": "image/jpeg",
    "filename": "Material-Callouts-c4655c0c.pDF",
    "data_source": {}
  }
}

For images of data plot graphs (for example bar charts, line charts, pie charts, and scatter plots), the text field will contain a type of graph; description containing a summary of the graph, data containing numerical values that can be used to regenerate the graph, and texts containing the text strings found in the graph. Here is an example. Line breaks have been inserted here for readability. The output will not contain these line breaks.

{
  "type": "Image",
  "element_id": "322b067931bc59555ae95dbb63ab9b78",
  "text": "{
    \"type\": \"graph\",
    \"description\": \"This is a comprehensive comparison chart showing 6 different 
      metrics across 7 machine learning models (RF, FFNN, LSTM, MHA, GCN, GCN+MHA, 
      MT+GCN+MHA) represented by different colored bars. The metrics include 
      PAVIS-TW-MSE, PAVIS-LS-MSE, PAVIS-EL@ACC 1, Trainable Parameters, Training 
      Time, and Inference Time. The models are categorized into paradigms: SNN (blue), 
      TNN (green), ReNN (orange), TReNN (purple), and MT-TReNN (red). Key trends show 
      RF and FFNN having higher MSE values but faster inference times, while more 
      complex models like MT+GCN+MHA achieve better accuracy but require more parameters 
      and training time.\", 
    \"data\": {
      \"PAVIS-TW-MSE\": {
        \"RF\": 1.222, \"FFNN\": 1.039, \"LSTM\": 0.817, \"MHA\": 0.790, \"GCN\": 0.800, 
          \"GCN+MHA\": 0.768, \"MT+GCN+MHA\": 0.760
      }, 
      \"PAVIS-LS-MSE\": {
        \"RF\": 0.763, \"FFNN\": 0.878, \"LSTM\": 0.649, \"MHA\": 0.611, \"GCN\": 0.596, 
        \"GCN+MHA\": 0.551, \"MT+GCN+MHA\": 0.549
      }, 
      \"PAVIS-EL@ACC 1\": {
        \"RF\": 79.54, \"FFNN\": 77.05, \"LSTM\": 80.69, \"MHA\": 82.10, \"GCN\": 83.39, 
        \"GCN+MHA\": 85.95, \"MT+GCN+MHA\": 85.98
      }, 
      \"Trainable Parameters\": {
        \"RF\": 328050, \"FFNN\": 237790, \"LSTM\": 264516, \"MHA\": 318941, \"GCN\": 221886, 
        \"GCN+MHA\": 324018, \"MT+GCN+MHA\": 99638
      }, 
      \"Training Time\": {
        \"RF\": 17968, \"FFNN\": 1447, \"LSTM\": 1985, \"MHA\": 2502, \"GCN\": 1620, 
        \"GCN+MHA\": 2808, \"MT+GCN+MHA\": 748
      }, 
      \"Inference Time\": {
        \"RF\": 190, \"FFNN\": 35, \"LSTM\": 75, \"MHA\": 79, \"GCN\": 69, 
        \"GCN+MHA\": 108, \"MT+GCN+MHA\": 18
      }
    }, 
    \"texts\": \"PAVIS - TW - MSE (↓) PAVIS - LS - MSE (↓) PAVIS - EL@ACC 1 (↑) Paradigm 
      SNN TNN ReNN TReNN MT-TReNN 1.222 1.039 0.817 0.790 0.800 0.768 0.760 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 0.763 0.878 0.649 0.611 0.596 0.551 0.549 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 79.54 77.05 80.69 82.10 83.39 85.95 85.98 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA Trainable Parameters (↓) Training Time (ms/epoch) (↓) 
      Inference Time (ms) (↓) 328050 237790 264516 318941 221886 324018 99638 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 17968 1447 1985 2502 1620 2808 748 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 190 35 75 79 69 108 18 RF FFNN 
      LSTM MHA GCN GCN+MHA MT+GCN+MHA\"
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 17,
    "image_base64": "/9j...<full results omitted for brevity>...//Z",
    "image_mime_type": "image/jpeg",
    "filename": "250713305v1-48da34f2.pdf",
    "data_source": {}
  }
}

The image_base64 field is generated only for documents or PDF pages that are partitioned by using the High Res strategy. This field is not generated for documents or PDF pages that are partitioned by using the Fast or VLM strategy.

For workflows that use chunking, note the following changes:

Each Image element is replaced by a CompositeElement element.
This CompositeElement element will contain the image’s summary description as part of the element’s text field.
This CompositeElement element will not contain an image_base64 field.

Here are three examples of the descriptions for detected images. These descriptions are generated with GPT-4o by OpenAI:

Description of an image showing a scatter plot graph

Description of an image showing the Matthews Correlation Coefficient for different VQA datasets

Description of an image showing three scatter plots

Any embeddings that are produced after these summaries are generated will be based on the text field’s contents.

Generate image descriptions

To have Unstructured generate image descriptions, do the following:

For Unstructured Pipelines users, add an Enrichment node of type NER to an Unstructured custom workflow.
For Unstructured API users, add an Image Description task. You add this task as either as an object in a workflow_nodes array (for curl) or as a WorkflowNode in a WorkflowNodes collection (for Python). This object or collection applies whenever you create a workflow, update a workflow, or create a workflow job that processes local files.

Structured data extractor

Enriching

Generate image descriptions

​Generate image descriptions

Generate image descriptions