[Bug]: Can't fetch the downloaded file

[mikurisan]

crawl4ai version

0.7.4

Expected Behavior

fetch the downloaded file using downloaded_files attribute.

Current Behavior

downloaded_files attr is None even though the target file has been downloaded successfully.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

async def download_detail_pdf(self, query_url: str):
        """Click "export" button to download the pdf file of current detail page."""
        wait_for_button = ".long-project-property-right > div:nth-child(3)"
        config1 = CrawlerRunConfig(
            session_id=self.session_id,
            wait_for=wait_for_button,
            js_only=True
        )
        await self.crawler.arun(query_url, config=config1)

        js_click_button = """
        const selector = ".long-project-property-right > div:nth-child(3)";
        const button = document.querySelector(selector);
        if (button) button.click();
        """
        config2 = CrawlerRunConfig(
            session_id=self.session_id,
            js_code=js_click_button,
            wait_for=1
        )
        result = await self.crawler.arun(query_url, config=config2)
        

        if result.downloaded_files:
            for file_path in result.downloaded_files:
                print("=====================")
                print(file_path)
        
        return None

OS

Windows

Python version

3.11

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

...
[SCRAPE].. ◆ https://qiye.qianlima.com/new_qd_yfbsite/#/infoC...archPage&isFirstZhaobiao=false&searchKeyWord=机器人  | ✓ | ⏱: 0.08s 
[COMPLETE] ● https://qiye.qianlima.com/new_qd_yfbsite/#/infoC...archPage&isFirstZhaobiao=false&searchKeyWord=机器人  | ✓ | ⏱: 0.35s 
====================================================
None
[FETCH]... ↓ https://qiye.qianlima.com/new_qd_yfbsite/#/infoC...archPage&isFirstZhaobiao=false&searchKeyWord=机器人  | ✓ | ⏱: 0.31s 
[SCRAPE].. ◆ https://qiye.qianlima.com/new_qd_yfbsite/#/infoC...archPage&isFirstZhaobiao=false&searchKeyWord=机器人  | ✓ | ⏱: 0.10s 
[COMPLETE] ● https://qiye.qianlima.com/new_qd_yfbsite/#/infoC...archPage&isFirstZhaobiao=false&searchKeyWord=机器人  | ✓ | ⏱: 0.41s 
[FETCH]... ↓ Downloading 国产智能化腔镜手术机器人整机制造及应用.pdf to D:/data\国产智能化腔镜手术机器人整机制造及应用.pdf 
...
[COMPLETE] ● Downloaded 国产智能化腔镜手术机器人整机制造及应用.pdf successfully 
...

[Ahmed-Tawfik94]

Hey @mikurisan ,

I see what's happening. It looks like you're using two separate arun calls, right? The trick is that the downloaded_files list is only on the result from the first call (the one that actually does the click).

It doesn't carry over to the second arun call, so when you check the result from that one, it's empty.

It's an easy fix! You just need to combine your actions and make sure your crawler is set up to handle downloads in the first place.

Here’s how you can set it all up:

1. The Setup (Do this once)

First, you need to tell the crawler to accept downloads and where to save them. You do this once when you create the crawler, using BrowserConfig.

from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler
import os

# This is your global download folder
download_path = os.path.join(os.getcwd(), "my_project_downloads")
os.makedirs(download_path, exist_ok=True)

# Set this up when you create your crawler
global_config = BrowserConfig(
    accept_downloads=True,
    downloads_path=download_path  # <-- set the global path here
)

# ... then create your crawler with it ...
# (This assumes 'self.crawler' is created like this somewhere in your class)
# self.crawler = AsyncWebCrawler(config=global_config)

---

2. The Fix (Your Function)

Now, your function just needs one arun call to do the click, wait, and get the file.

async def download_detail_pdf(self, query_url: str):
    """
    Clicks 'export' AND fetches the file path in one single action.
    """
    
    # The JS to click the button
    js_to_click_button = """
    const selector = ".long-project-property-right > div:nth-child(3)";
    const button = document.querySelector(selector);
    if (button) button.click();
    """
    
    # Just one config for this one action
    action_config = CrawlerRunConfig(
        session_id=self.session_id,
        js_code=js_to_click_button,
        wait_for=10  # Give it time to download
    )
    
    # Run the single 'arun' call
    result = await self.crawler.arun(query_url, config=action_config)

    # Now check the result from THIS call 👍
    if result.downloaded_files:
        print("Success! Got the files:")
        for file_path in result.downloaded_files:
            print(file_path)
        return result.downloaded_files
    else:
        print("No files were downloaded in this step.")
        return None

You can read more about it in the docs too.

Hope that helps!

[mikurisan]

@Ahmed-Tawfik94

Thank you very much.

What puzzles me is that the first arun was only to wait for the "click button" selector to appear, and the second arun was actually clicking the "click button" and triggering the download, yet I failed to retrieve the result of the second arun.

The reason for doing this seems to be that in the Crawl4AI framework, the js_code is executed first, and then wait_for is called. I just wanted to ensure the button appears first.