> ## Documentation Index
> Fetch the complete documentation index at: https://docs.unstructured.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Firecrawl
[Firecrawl](https://firecrawl.dev/) is a tool that allows you to crawl websites and extract data from them.
You can use Unstructured to process the data that Firecrawl extracts from these websites. Unstructured turns this extracted
website data into a single set of standard [Unstructured document elements and metadata](/ui/document-elements),
improving the accuracy, relevance, and performance of your RAG applications, agentic AI workflows, model fine tuning, and more.
This example provides a hands-on walkthrough of how to use Firecrawl with Unstructured. In this example, you use Python code to have Firecrawl
extract data from a ficticious website and then add the extracted data to an Amazon S3 bucket within your AWS account. From there, you use the Unstructured user interface (UI) to set up an
ETL+ workflow that takes the extracted data from the S3 bucket, converts it into Unstructured document elements and metadata, adds
vector embeddings, and then sends the results over into a serverless index within a Pinecone vector store.
## Requirements
To complete this walkthrough, you must first have:
* An Unstructured account.
* A Firecrawl account and a Firecrawl API key for that account.
* An AWS account and a properly configured Amazon S3 bucket with bucket access credentials within that account.
* A Pinecone account and a Pinecone API key for that account.
* Python and the necessary development toolchain installed on your local development machine.
The following sections describe how to get these requirements.
### Unstructured account
Before you begin, you must have an Unstructured account and an Unstructured API key, as follows:
1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).
To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).
For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
3. Get your Unstructured API key:
a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.
For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
for. Each API key works with one and only one organizational workspace. [Learn more](/ui/account/workspaces#create-an-api-key-for-a-workspace).
b. Click **Generate API Key**.
c. Follow the on-screen instructions to finish generating the key.
d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.
### Firecrawl account and API key
Before you begin, you must have a Firecrawl account and a Firecrawl API key, as follows:
1. [Sign up or sign in to your Firecrawl account](https://www.firecrawl.dev/signin/signup).
2. After you sign in, [get your Firecrawl API key](https://www.firecrawl.dev/app/api-keys).
### Amazon S3 bucket with access permissions and credentials
This walkthrough uses an Amazon S3 bucket as the Unstructured ETL+ workflow's source. So, before you begin, you must have an AWS account, and within that account, a properly configured Amazon S3 bucket with bucket access credentials
consisting of an AWS access key ID and secret access key for the AWS IAM user that has access to the bucket.
Follow the S3 connector instructions to
[create and configure the bucket and get the bucket access credentials](/ui/sources/s3) if you do not already have this all set up. (In these
instructions, do not follow the directions to use the Unstructured UI to create the S3 source connector. You will do this later when you create the Unstructured ETL+ workflow.)
### Pinecone account and API key
Before you begin, you must have a Pinecone account and a Pinecone API key, as follows:
1. [Sign up](https://app.pinecone.io/?sessionType=signup) or [log in](https://app.pinecone.io/?sessionType=login) to your Pinecone account.
2. After you log in, [get your Pinecone API key](https://docs.pinecone.io/guides/projects/manage-api-keys#create-an-api-key).
### Python and development toolchain installation
Before you can start coding on your local machine, you must install Python, and you should also install a Python package and project manager to manage your project's Firecrawl, AWS, Pinecone, and other code dependencies.
This walkthrough uses the popular Python package and project manager [uv](https://docs.astral.sh/uv/) (although `uv` is not required to use Firecrawl, AWS, or Pinecone with Python).
To install `uv`, run one of the following commands, depending on your operating system:
To use `curl` with `sh`:
```bash theme={null}
curl -fsSL https://get.uv.dev | bash
```
To use `wget` with `sh` instead:
```bash theme={null}
wget -qO- https://astral.sh/uv/install.sh | sh
```
To use PowerShell with `irm` to download the script and run it with `iex`:
```powershell theme={null}
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```
If you need to install `uv` by using other approaches such as PyPI, Homebrew, or WinGet,
see [Installing uv](https://docs.astral.sh/uv/getting-started/installation/).
This walkthrough should work with Python 3.9 or later and has been tested with Python 3.12.
`uv` will detect and use Python if you already have it installed.
To view a list of installed Python versions, run the following command:
```bash theme={null}
uv python list
```
If, however, you do not already have Python installed, you can install a version of Python for use with `uv`
by running the following command. For example, this command installs Python 3.12 for use with `uv`:
```bash theme={null}
uv python install 3.12
```
Switch to the directory on your local development machine where you want to create the project directory for this walkthrough.
This example creates a project directory named `firecrawl_unstructured_demo` within your current working directory and then
switches to this new project directory:
```bash theme={null}
mkdir firecrawl_unstructured_demo
cd firecrawl_unstructured_demo
```
From within the new project directory, use `uv` to initialize the project by running the following command:
```bash theme={null}
uv init
```
To isolate and manage your project's code dependencies, you should create a virtual environment. This walkthrough uses
the popular Python virtual environment manager [venv](https://docs.python.org/3/library/venv.html) (although `venv` is not required to use CrewAI or the Unstructured API's workflow operations MCP Server).
From the root of your project directory, use `uv` to create a virtual environment with `venv` by running the following command:
```bash theme={null}
# Create the virtual environment by using the current Python version:
uv venv
# Or, if you want to use a specific Python version:
uv venv --python 3.12
```
To activate the `venv` virtual environment, run one of the following commands from the root of your project directory:
* For `bash` or `zsh`, run `source .venv/bin/activate`
* For `fish`, run `source .venv/bin/activate.fish`
* For `csh` or `tcsh`, run `source .venv/bin/activate.csh`
* For `pwsh`, run `.venv/bin/Activate.ps1`
* For `cmd.exe`, run `.venv\Scripts\activate.bat`
* For `PowerShell`, run `.venv\Scripts\Activate.ps1`
If you need to deactivate the virtual environment at any time, run the following command:
```bash theme={null}
deactivate
```
You will use the Firecrawl Python SDK later to extract data from a website.
Use `uv` to install the Firecrawl Python SDK package, by running the following command:
```bash theme={null}
uv add firecrawl-py
```
You will use the AWS SDK for Python later to have Firecrawl add the extracted data to your Amazon S3 bucket.
1. Use `uv` to install the AWS SDK for Python package, by running the following command:
```bash theme={null}
uv add boto3
```
2. To enable the AWS SDK for Python to authenticate with your AWS account, create a file named `credentials` in the `~/.aws/` directory for macOS or Linux, or the `:\Users\\.aws\` directory for Windows. Then add the
AWS access key ID and secret access key of the AWS IAM user that has access to your Amazon S3 bucket, and the [short code for the AWS Region](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) of the bucket (for example, `us-east-1`), to the `credentials` file.
In the following `credentials` file example, replace the following placeholders:
* Replace `` with the AWS access key ID of the AWS IAM user that has access to the bucket.
* Replace `` with the secret access key for the related access key ID.
* Replace `` with the short code for the AWS Region of the bucket.
```text theme={null}
[default]
aws_access_key_id =
aws_secret_access_key =
region =
```
You will use the Pinecone Python SDK later to create the Pinecone serverless index and, in an optional step, to help enable RAG chat on the data that will be storedin the index.
Use `uv` to install the Pinecone Python SDK package, along with the `grpc` extra to enable the programmatic creation of a Pinecone serverless index later, by running the following command:
```bash theme={null}
uv add "pinecone[grpc]"
```
## Step 1: Extract the website data and add the data to the bucket
In this step, you use Firecrawl to extract data from a website and then add the extracted data to the Amazon S3 bucket.
1. Set your Firecrawl API key in your project as an environment variable for your code to use. To do this:
a. In the root of your code project, create a file named `.env`.
b. Add the following environment variable to this `.env` file, and replace
`` with your Firecrawl API key:
```text theme={null}
FIRECRAWL_API_KEY=
```
2. To enable your Firecrawl code to access the environment variable, use `uv` to install the `dotenv` package by running the following command:
```bash theme={null}
uv add dotenv
```
3. In the `.gitignore` file in the root of your project, add the following line, to help prevent
accidentally checking in your Firecrawl API key (or anything else in the `.env` file) into any shared code repositories later:
```text theme={null}
.env
```
1. In the `.env` file, add the following line, which defines an environment variable representing the base URL
of the website to crawl. This walkthrough uses a website named [Books to Scrape](https://books.toscrape.com/) that contains ficticious data (although you can use any accessible website you want):
```text theme={null}
FIRECRAWL_BASE_URL=https://books.toscrape.com/
```
2. Also in the `.env` file, add the following line, which defines an environment variable representing the name of the target Amazon S3 bucket to
have Firecrawlstore the website crawl results in. Replace `` with the name of the bucket:
```text theme={null}
FIRECRAWL_S3_BUCKET_NAME=
```
Enter only the name of the bucket. Do not prefix the bucket's name with `s3://`. Do not include any trailing slash (`/`) after the bucket name.
3. Create a file named `firecrawl_extract.py` in the root of your project directory, and add the following code to it:
```python theme={null}
from firecrawl import FirecrawlApp, ScrapeOptions
from dotenv import load_dotenv
import os, boto3, json
from datetime import datetime
load_dotenv()
def save_to_s3(crawl_result, bucket_name, prefix):
s3 = boto3.client("s3")
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
# Save the full crawl results to the bucket as a single JSON file.
full_results_key = f"{prefix}/{timestamp}/full_results.json"
s3.put_object(
Bucket=bucket_name,
Key=full_results_key,
Body=json.dumps(crawl_result.model_dump(), indent=2, default=str),
)
# Save individually crawled pages to the bucket as HTML files.
for idx, page in enumerate(crawl_result.data):
if page.html is not None:
page_key = f"{prefix}/{timestamp}/pages/{idx}.html"
s3.put_object(Bucket=bucket_name, Key=page_key, Body=page.html)
print(f"Successfully saved {idx}.html to {bucket_name}/{page_key}")
print(f"Successfully saved {len(crawl_result.data)} pages to {bucket_name}/{full_results_key}")
def main():
app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))
result = app.crawl_url(
url=os.getenv("FIRECRAWL_BASE_URL"),
limit=5, # Maximum number of pages to crawl.
scrape_options=ScrapeOptions(
formats=["html"], # Save the crawled pages as HTML files.
waitFor=1000, # Wait for one second before crawling the currentpage.
timeout=10000 # Move on to the next page if no response within 10 seconds.
)
)
save_to_s3(
crawl_result=result,
bucket_name=os.getenv("FIRECRAWL_S3_BUCKET_NAME"),
prefix="crawls"
)
if __name__ == "__main__":
main()
```
The preceding code does the following:
* Loads from your `.env` file your Firecrawl API key, the base URL for Firecrawl to use for website crawling, and the S3 bucket name for Firecrawl to send the website crawl results to.
* Calls a function named `main` that uses Firecrawl to crawl the website.
* The `main` function then calls the `save_to_s3` function, which adds the website crawl results to the S3 bucket.
You can change the where the website crawl results are saved in your S3 bucket by changing the `full_results_key` and `page_key` variables. For example, you might want to save the
website crawl results in a single folder and then keep overwriting those results with new results as they come in, instead of adding new results to separate subfolders.
You can also change the number of crawled pages by changing the `limit` argument, outputting the results as Markdown instead of HTML, and so on. For more
information, see the [Firecrawl Python SDK documentation](https://docs.firecrawl.dev/sdks/python).
From the root of your project, use `uv` to run the script to extract the data from the website by running the following command:
```bash theme={null}
uv run firecrawl_extract.py
```
After the script runs, you should see the following output, where `` is the name of your bucket, and `` is the timestamp generated by the script:
```bash theme={null}
Successfully saved 0.html to /crawls//pages/0.html
Successfully saved 1.html to /crawls//pages/1.html
Successfully saved 2.html to /crawls//pages/2.html
Successfully saved 3.html to /crawls//pages/3.html
Successfully saved 4.html to /crawls//pages/4.html
Successfully saved 5 pages to /crawls//full_results.json
```
To verify that the data was successfully added, open your AmazonS3 console
and look in the `crawls` folder within your bucket.
## Step 2: Create a Pinecone serverless index
In this step, you create a Pinecone serverless index that Unstructured will use to store its results based on the
extracted website data that Firecrawl sends to your Amazon S3 bucket.
Set your Pinecone API key as an environment variable for your code to use. To do this, add the following environment variable to your `.env` file, and replace
`` with your Pinecone API key:
```text theme={null}
PINECONE_API_KEY=
```
Add the following line to your `.env` file, and replace `` with the name of the serverlessindex you want to create:
```text theme={null}
PINECONE_INDEX_NAME=
```
Create a file named `pinecone_create_index.py` in the root of your project directory and add the following code to it:
```python theme={null}
from pinecone.grpc import PineconeGRPC as Pinecone
from pinecone import ServerlessSpec
import os
from dotenv import load_dotenv
load_dotenv()
pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
pc.create_index(
name=os.getenv("PINECONE_INDEX_NAME"),
vector_type="dense",
dimension=3072,
metric="cosine",
spec=ServerlessSpec(
cloud="aws",
region="us-east-1"
)
)
```
The preceding code creates in your Pinecone account a serverless index with the specified name for the specified cloud provider and within the specified region for that provider.
The index is configured to accept vectors with 3072 dimensions with a cosine metric and a dense vector type.
From the root of your project, use `uv` to run the script to create the Pinecone serverless index by running the following command:
```bash theme={null}
uv run pinecone_create_index.py
```
After the script runs, to verify that the index was successfully added, log in to your Pinecone account. Then open your default project,
and click **Indexes** under **Database** in the sidebar to view the new index that was added.
## Step 3: Set up and run the Unstructured ETL+ workflow
In this step, you set up and run the Unstructured ETL+ workflow that will take the extracted website data from your Amazon S3 bucket, process it to turn it into
Unstructured elements and metadata, add vector embeddings to the elements, and then add all of this information to your Pinecone serverless index.
1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **Connectors**.
3. With **Sources** selected, **New**.
4. Enter some name for the source connector, such as `s3-firecrawl-source`.
5. For **Provider**, click **Amazon S3**.
6. Click **Continue**.
7. For **Bucket URI**, enter `s3:///crawls//pages/`, replacing `` with the name of your bucket, and `` with the timestamp generated by the script in the previous step..
8. For **Authentication Method**, select **Secret**.
9. Enter your AWS access key ID and secret access key.
10. Click **Continue**.
11. Click **Save and Test**.
1. On the sidebar, click **Connectors**.
2. With **Destinations** selected, **New**.
3. Enter some name for the destination connector, such as `pinecone-firecrawl-destination`.
4. For **Provider**, click **Pinecone**.
5. Click **Continue**.
6. For **Index Name**, enter the name of the Pinecone index you created earlier, such as `firecrawl-dense-index`.
7. Leave **Namespace** blank. Unstructured will automatically use the default namespace in the index.
8. Leave **Batch Size** set to the default value of `50`.
9. For **API Key**, enter your Pinecone API key.
10. Click **Save and Test**.
1. On the sidebar, click **Workflows**.
2. Click **New Workflow**.
3. With **Build it Myself** selected, click **Continue**.
4. In the workflow designer, click the **Source** workflow node.
5. In the node settings pane, on the **Connectors** tab, select the name of the source connector you created earlier, such as `s3-firecrawl-source`.
6. Click the **Destination** workflow node.
7. In the node settings pane, on the **Connectors** tab, select the name of the destination connector you created earlier, such as `pinecone-firecrawl-destination`.
8. Between the **Partitioner** and **Destination** nodes, click the **+** (add node) button.
9. Click **Transform > Embedder**.
10. Click the **Embedder** workflow node.
11. In the **Settings** pane, under **Azure OpenAI**, select **text-embedding-3-large**.
12. Click any blank space in the workflow designer.
13. In the workflow settings pane, click **Details**.
14. For **Name**, click the pencil (edit) icon and then enter some name for the workflow, such as `firecrawl-s3-to-pinecone-workflow`.
Then click the checkmark (save) icon.
15. Switch **Active** to on.
16. Click **Save**.
1. On the sidebar, click **Workflows**.
2. Next to the name of the workflow you created earlier, click **Run**.
1. On the sidebar, click **Jobs**.
2. Click the name of the workflow you created earlier.
3. Monitor the job's progress in the job details pane until the status changes to **Finished**.
1. One way to view the results is to create a file named `pinecone_fetch_from_index.py` in the root of your project directory and add the following code to it:
```python theme={null}
from pinecone import Pinecone
import os
from dotenv import load_dotenv
load_dotenv()
pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
index = pc.Index(os.getenv("PINECONE_INDEX_NAME"))
ids = []
# The list_paginated() method returns a dict with 'vectors' key
paginated_results = index.list_paginated(namespace="default")
for page in paginated_results['_data_store']['vectors']:
ids.append(page['id'])
response = index.fetch(
ids=ids,
namespace="default"
)
for vector_id, vector_data in response["vectors"].items():
print(f"ID: {vector_id}\nMetadata: {vector_data.get('metadata')}\n\n")
```
2. From the root of your project, use `uv` to run the script as follows:
```bash theme={null}
uv run pinecone_fetch_from_index.py
```
The output will show each record's ID and metadata. For brevity and performance,
the records' vector embeddings are not included in the output.
## Additional resources
* [Mastering Firecrawl's Crawl Endpoint: A Complete Web Scraping Guide](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl)