[NODRIVER] Getting detected by anti-bot when running on a VPS or headless

[thegeosman]

Have been attempting to automate consults in a website to paste a code number, click consult, get the PDF which are shown there. Very simple process.

It runs amazingly in my local desktop with normal run (with the opening of the browser), run locally using linux/python xvfbwrapper package hiding the screen.

However when running in a docker container in VPS (which is my necessity) it doesn't work and get detected, as the launch of the browser and start of the get of the initial domain.

I have been using random delays between steps of the processing.

I have used small number of repetitions per launched each container.

Each container use different domain IP.

We try proxies from free proxies and paid residential proxies to test. No proxies worked.

What more can I twick to try to fix the anti-bot triggering?

[devblack]

Screen detection

[ultrafunkamsterdam]

Datacenter IP are all known by these protectiona . As well as free proxies. Furthermore, when running docker on server, all containers using the host main ip unless you do some iptables

[thegeosman]

Screen detection

Technically by using linux/python xvfbwrapper to hide the screen it works at my local computer, but do not work in the VPS server. There is any other config possible in NODRIVER to avoid it?

[thegeosman]

Datacenter IP are all known by these protectiona . As well as free proxies. Furthermore, when running docker on server, all containers using the host main ip unless you do some iptables

I'm looking into it to see a way to do it. But there is any way that NODRIVER can be configured to solve it?

[devblack]

Screen detection

Technically by using linux/python xvfbwrapper to hide the screen it works at my local computer, but do not work in the VPS server. There is any other config possible in NODRIVER to avoid it?

Can you shared a sample of your script. To debug the problem

[thegeosman]

Screen detection

Technically by using linux/python xvfbwrapper to hide the screen it works at my local computer, but do not work in the VPS server. There is any other config possible in NODRIVER to avoid it?

Can you shared a sample of your script. To debug the problem

Absolutely.

This script runs perfectly on local desktop or notebook using the Xvfb (to hide the screen) or not. Once it goes in a container or in the VPS (in aws or google) it doesn't work.

I try this same code with paid residential proxies (socks5) and with free proxies (socks5) too. None worked, all got detected.

`import logging
import os
import glob
import time
import nodriver as uc
from datetime import datetime

logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)

logger = logging.getLogger(name)
os.environ["XDG_SESSION_TYPE"] = "x11"

class CIBDownloader:

 """Class to handle CIB certificate downloading automation"""
 def __init__(self):
     self.browser = None
     self.tab = None
     self.download_path = os.path.join(os.getcwd(), "downloads")

 async def setup_browser(self, headless=False):

     """Initialize the browser with appropriate settings"""
     try:
         # Create download directory if it doesn't exist
         os.makedirs(self.download_path, exist_ok=True)
         logger.info(f"Download directory: {self.download_path}")
         browser_args_list = [
             '--start-maximized',
             # '--disable-extensions',
             '--disable-plugins',
             '--disable-dev-shm-usage',
             '--disable-gpu',
             '--no-first-run',
             '--disable-default-apps',
             '--disable-popup-blocking'
         ]

         # Configure browser
         config = uc.Config( headless=headless, browser_args=browser_args_list)
         # Start browser
         self.browser = await uc.start(config)
         logger.info("Browser started successfully")
         return True
     except Exception as e:
         logger.error(f"Failed to setup browser: {str(e)}")
         return False

 async def navigate_to_cib_page(self, url="https://servicos.receitafederal.gov.br/servico/certidoes/#/home/cib"):
     """Navigate to the CIB service page"""
     try:
         logger.info(f"Navigating to: {url}")

         self.tab = await self.browser.get("https://servicos.receitafederal.gov.br/servico/certidoes/#/home/cib")
         # Wait for page to load
         await self.tab.sleep(5)
         logger.info("Page loaded successfully")
         return True
     except Exception as e:
         logger.error(f"Failed to navigate to CIB page: {str(e)}")
         return False

 async def handle_cookies(self):
     """Handle cookie consent banner if present"""
     try:
         logger.info("Checking for cookie consent banner...")
         time.sleep(1)
         # Common cookie consent button texts in Portuguese
         cookie_text = "aceitar"

         try:
             cookie_btn = await self.tab.find(cookie_text, timeout=2, best_match=True)
             if cookie_btn:
                 logger.info(f"Found cookie consent button with text: '{cookie_text}'")
                 await cookie_btn.click()
                 await self.tab.sleep(1)
                 logger.info("Cookie consent accepted")
                 return True
         except Exception:
             pass

         logger.info("No cookie consent banner found or already handled")
         return True

     except Exception as e:
         logger.warning(f"Error handling cookies (continuing anyway): {str(e)}")
         return True

 async def enter_cib_code(self, cib_code=""):
     """Enter the CIB code in the input field"""
     try:
         logger.info(f"Entering CIB code: {cib_code}")

         # Wait for the page to fully load
         await self.tab.sleep(2)

         # Common input field selectors for CIB
         selector = "input[placeholder*='Informe o CIB']"
         cib_input = None

         try:
             # cib_input = self.tab.find(selector, timeout=3)
             cib_input =  await self.tab.select(selector, timeout=3)
             if cib_input:
                 logger.info(f"Found CIB input field with selector: {selector}")
         except Exception:
             pass
         print(" waiting... ")
         time.sleep(1)
         # Clear and enter the CIB code
         await cib_input.click()
         print(" waiting... 1 ")
         time.sleep(1)
         await cib_input.send_keys(cib_code)
         print(" waiting... 2 ")
         time.sleep(1)

         logger.info("CIB code entered successfully")
         return True

     except Exception as e:
         logger.error(f"Failed to enter CIB code: {str(e)}")
         return False

 async def enter_initial_date(self):
     """Enter the initial date in the date field"""
     try:
         logger.info(f"Entering initial date: {'15/08/2019'}")

         date_element = []
         # Look for date input fields
         selector = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-informar-parametro-cib/app-informar-parametro/form/div[1]/div/div[2]/div[1]/br-date-picker/div/div[1]/input"
         try:
             date_input = await self.tab.xpath(selector)
             print(" date_input ", date_input)
             date_element = date_input[0]
             if date_input:
                 logger.info(f"Found date input field with selector: {selector}")
                 logger.info(f"Found date input field with date_element: {date_element}")
         except Exception:
             pass

         if date_element:
             print(" date_element ", date_element)
             await date_element.click()
             await date_element.clear_input()
             await date_element.send_keys("15/08/2019")

             """ Make sure the new date is there"""
             await date_element.send_keys("\r")
             logger.info("Initial date entered successfully")

             return True
         else:
             pass

     except Exception as e:
         logger.error(f"Failed to enter initial date: {str(e)}")
         return False

 async def click_consult_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Consultar Certidão' button...")

         # Try to find the button by text
         button_text = "Consultar Certidão"

         consult_btn = None
         try:
             consult_btn = await self.tab.find(button_text, timeout=3, best_match=True)
             if consult_btn:
                 logger.info(f"Found button with text: '{button_text}'")
         except Exception:
             pass

         if consult_btn is not None:
             # Click the button
             await consult_btn.click()
             await self.tab.sleep(3)  # Wait for response

             logger.info("Consultation button clicked successfully")
             return True

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def click_consult_second_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for ' Consultar Certidão ' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-informar-parametro-cib/app-informar-parametro/form/div[2]/button"
         consult_btn = ''
         try:
             consult_btn = await self.tab.xpath(button_xpath, timeout=3)
             if consult_btn:
                 consult_btn = consult_btn[0]
                 logger.info(f"Found button with text: '{button_xpath}'")
         except Exception:
             pass

         print(" consult_btn --- ", consult_btn)

         # if consult_btn is not None:
         if consult_btn:
             # Click the button
             await consult_btn.mouse_move()
             await consult_btn.click()
             logger.info("Consultation button clicked successfully")
             return True

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def download_certificate(self, cib_code=""):
     """Download the certificate if available"""
     try:
         logger.info("Looking for download link/button...")
         time.sleep(1)

         download_link = []
         try:
             download_link = await self.tab.query_selector_all("i[class='fas fa-download']")
             print(" download_link ", len(download_link), " --- ", download_link)
             if download_link:
                 logger.info(f"Found download link with text: '{download_link}'")
         except Exception:
             pass

         if download_link:
             # Set download behavior for the browser
             await self.tab.set_download_path(self.download_path)

             list_existing_files = glob.glob(os.path.join(self.download_path,"*"+cib_code+"*.pdf"))
             file_add_name = 1
             if len(list_existing_files)>0:
                 file_add_name = max([int(i.split(os.sep)[-1].split(".pdf")[0].split(cib_code)[-1].strip().split("_")[-1]) for i in list_existing_files if i.split(os.sep)[-1].split(".pdf")[0].split(cib_code)[-1].strip().split("_")[-1]])
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             down_counter =  1
             for d_link in download_link:
                 # Click the download link
                 try:
                     await self.tab.sleep(1)
                     time.sleep(1)

                     down_xpath_link = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[2]/div[2]/br-table/div/ngx-datatable/div/div/datatable-body/datatable-selection/datatable-scroller/datatable-row-wrapper[%s]/datatable-body-row/div[2]/datatable-body-cell[7]/div/div/button/i" % str(down_counter)

                     try:
                         download_link = await self.tab.xpath(down_xpath_link, timeout=3)
                         if download_link:
                             download_link_item = download_link[0]
                             await download_link_item.click()
                             logger.info(f"Found download link with text: '{down_xpath_link}'")
                     except Exception:
                         pass

                     time.sleep(1)
                     logger.info(f"Download initiated. File should be saved to: {self.download_path}")
                     file_name_str = os.path.join(self.download_path,"Certidao-"+cib_code+".pdf")
                     if os.path.isfile(file_name_str):

                         os.rename(file_name_str, file_name_str.replace(".pdf","_"+str(file_add_name)+".pdf"))
                     file_add_name += 1
                     down_counter += 1
                 except Exception as down_error:
                     print(" dow_error : ", down_error)
                     await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             return True
         else:
             logger.warning("No download link found. The certificate may not be available.")
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             return False

     except Exception as e:
         logger.error(f"Failed to download certificate: {str(e)}")
         return False

 async def click_return_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Voltar' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[4]/div[1]/app-voltar/button"

         try:
             return_button_element = await self.tab.xpath(button_xpath, timeout=3)
             if return_button_element:
                 return_button_element_item = return_button_element[0]
                 await return_button_element_item.click()
                 logger.info(f"Found return button with xpath: '{button_xpath}'")

                 return True
         except Exception:
             pass

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def click_new_consult_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Nova Consulta' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[4]/div[2]/button"

         try:
             return_button_element = await self.tab.xpath(button_xpath, timeout=3)
             # print(" return_button_element ", len(return_button_element), " --- ", return_button_element)
             if return_button_element:
                 return_button_element_item = return_button_element[0]
                 await return_button_element_item.click()
                 logger.info(f"Found New consult button with xpath: '{button_xpath}'")
                 # time.sleep(1)
                 return True
         except Exception:
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def run(self, cib_code="", initial_date="15/08/2019", headless=False):
     """Main method to run the complete workflow"""
     # try:
     logger.info("Starting CIB certificate download process...")

     # Setup browser
     await self.setup_browser(headless=headless)

     # Navigate to CIB page
     await self.navigate_to_cib_page()

     time.sleep(1)
     # Handle cookies
     await self.handle_cookies()
     time.sleep(1)
     # Enter CIB code
     await self.enter_cib_code(cib_code)
     time.sleep(1)
     # Click consultation button
     await self.click_consult_button()
     time.sleep(1)
     # Enter initial date
     await self.enter_initial_date()
     time.sleep(1)
     # Download certificate
     success = await self.download_certificate(cib_code)

     if success:
         logger.info("CIB certificate download process completed successfully!")
     else:
         logger.warning("Process completed but certificate may not have been downloaded")

     new_consult_button = await self.click_new_consult_button()

     # print(" new_consult_button --- ", new_consult_button)
     if new_consult_button:
         logger.info("Returned to the initial page successfully!")
     else:
         logger.warning("Error in the return")
     await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)
     return success

from xvfbwrapper import Xvfb

async def main():

 vdisplay = Xvfb()
 vdisplay.start()
 """Main entry point"""
 # Initialize the downloader
 downloader = CIBDownloader()

 # Configuration
 # '05325196'    '07447833' '45352860' '72670908' "83033467"
 CIB_CODE = "72670908"
 INITIAL_DATE = "15/08/2019"
 HEADLESS = False  # Set to True to run without GUI

 # Run the download process
 success = await downloader.run(
     cib_code=CIB_CODE,
     initial_date=INITIAL_DATE,
     headless=HEADLESS
 )
 vdisplay.stop()

if name == "main":
# Run the script
try:
# Use nodriver's event loop
uc.loop().run_until_complete(main())
except KeyboardInterrupt:
print("\n⚠️ Process interrupted by user")
except Exception as e:
print(f"❌ Unexpected error: {str(e)}")
logging.error(f"Unexpected error: {str(e)}", exc_info=True) `

[thegeosman]

Screen detection

Technically by using linux/python xvfbwrapper to hide the screen it works at my local computer, but do not work in the VPS server. There is any other config possible in NODRIVER to avoid it?

Can you shared a sample of your script. To debug the problem

Absolutely.

This script runs perfectly on local desktop or notebook using the Xvfb (to hide the screen) or not. Once it goes in a container or in the VPS (in aws or google) it doesn't work.

I try this same code with paid residential proxies (socks5) and with free proxies (socks5) too. None worked, all got detected.

`import logging import os import glob import time import nodriver as uc from datetime import datetime

logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s' )

logger = logging.getLogger(name) os.environ["XDG_SESSION_TYPE"] = "x11"

class CIBDownloader:

 """Class to handle CIB certificate downloading automation"""
 def __init__(self):
     self.browser = None
     self.tab = None
     self.download_path = os.path.join(os.getcwd(), "downloads")

 async def setup_browser(self, headless=False):

     """Initialize the browser with appropriate settings"""
     try:
         # Create download directory if it doesn't exist
         os.makedirs(self.download_path, exist_ok=True)
         logger.info(f"Download directory: {self.download_path}")
         browser_args_list = [
             '--start-maximized',
             # '--disable-extensions',
             '--disable-plugins',
             '--disable-dev-shm-usage',
             '--disable-gpu',
             '--no-first-run',
             '--disable-default-apps',
             '--disable-popup-blocking'
         ]

         # Configure browser
         config = uc.Config( headless=headless, browser_args=browser_args_list)
         # Start browser
         self.browser = await uc.start(config)
         logger.info("Browser started successfully")
         return True
     except Exception as e:
         logger.error(f"Failed to setup browser: {str(e)}")
         return False

 async def navigate_to_cib_page(self, url="https://servicos.receitafederal.gov.br/servico/certidoes/#/home/cib"):
     """Navigate to the CIB service page"""
     try:
         logger.info(f"Navigating to: {url}")

         self.tab = await self.browser.get("https://servicos.receitafederal.gov.br/servico/certidoes/#/home/cib")
         # Wait for page to load
         await self.tab.sleep(5)
         logger.info("Page loaded successfully")
         return True
     except Exception as e:
         logger.error(f"Failed to navigate to CIB page: {str(e)}")
         return False

 async def handle_cookies(self):
     """Handle cookie consent banner if present"""
     try:
         logger.info("Checking for cookie consent banner...")
         time.sleep(1)
         # Common cookie consent button texts in Portuguese
         cookie_text = "aceitar"

         try:
             cookie_btn = await self.tab.find(cookie_text, timeout=2, best_match=True)
             if cookie_btn:
                 logger.info(f"Found cookie consent button with text: '{cookie_text}'")
                 await cookie_btn.click()
                 await self.tab.sleep(1)
                 logger.info("Cookie consent accepted")
                 return True
         except Exception:
             pass

         logger.info("No cookie consent banner found or already handled")
         return True

     except Exception as e:
         logger.warning(f"Error handling cookies (continuing anyway): {str(e)}")
         return True

 async def enter_cib_code(self, cib_code=""):
     """Enter the CIB code in the input field"""
     try:
         logger.info(f"Entering CIB code: {cib_code}")

         # Wait for the page to fully load
         await self.tab.sleep(2)

         # Common input field selectors for CIB
         selector = "input[placeholder*='Informe o CIB']"
         cib_input = None

         try:
             # cib_input = self.tab.find(selector, timeout=3)
             cib_input =  await self.tab.select(selector, timeout=3)
             if cib_input:
                 logger.info(f"Found CIB input field with selector: {selector}")
         except Exception:
             pass
         print(" waiting... ")
         time.sleep(1)
         # Clear and enter the CIB code
         await cib_input.click()
         print(" waiting... 1 ")
         time.sleep(1)
         await cib_input.send_keys(cib_code)
         print(" waiting... 2 ")
         time.sleep(1)

         logger.info("CIB code entered successfully")
         return True

     except Exception as e:
         logger.error(f"Failed to enter CIB code: {str(e)}")
         return False

 async def enter_initial_date(self):
     """Enter the initial date in the date field"""
     try:
         logger.info(f"Entering initial date: {'15/08/2019'}")

         date_element = []
         # Look for date input fields
         selector = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-informar-parametro-cib/app-informar-parametro/form/div[1]/div/div[2]/div[1]/br-date-picker/div/div[1]/input"
         try:
             date_input = await self.tab.xpath(selector)
             print(" date_input ", date_input)
             date_element = date_input[0]
             if date_input:
                 logger.info(f"Found date input field with selector: {selector}")
                 logger.info(f"Found date input field with date_element: {date_element}")
         except Exception:
             pass

         if date_element:
             print(" date_element ", date_element)
             await date_element.click()
             await date_element.clear_input()
             await date_element.send_keys("15/08/2019")

             """ Make sure the new date is there"""
             await date_element.send_keys("\r")
             logger.info("Initial date entered successfully")

             return True
         else:
             pass

     except Exception as e:
         logger.error(f"Failed to enter initial date: {str(e)}")
         return False

 async def click_consult_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Consultar Certidão' button...")

         # Try to find the button by text
         button_text = "Consultar Certidão"

         consult_btn = None
         try:
             consult_btn = await self.tab.find(button_text, timeout=3, best_match=True)
             if consult_btn:
                 logger.info(f"Found button with text: '{button_text}'")
         except Exception:
             pass

         if consult_btn is not None:
             # Click the button
             await consult_btn.click()
             await self.tab.sleep(3)  # Wait for response

             logger.info("Consultation button clicked successfully")
             return True

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def click_consult_second_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for ' Consultar Certidão ' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-informar-parametro-cib/app-informar-parametro/form/div[2]/button"
         consult_btn = ''
         try:
             consult_btn = await self.tab.xpath(button_xpath, timeout=3)
             if consult_btn:
                 consult_btn = consult_btn[0]
                 logger.info(f"Found button with text: '{button_xpath}'")
         except Exception:
             pass

         print(" consult_btn --- ", consult_btn)

         # if consult_btn is not None:
         if consult_btn:
             # Click the button
             await consult_btn.mouse_move()
             await consult_btn.click()
             logger.info("Consultation button clicked successfully")
             return True

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def download_certificate(self, cib_code=""):
     """Download the certificate if available"""
     try:
         logger.info("Looking for download link/button...")
         time.sleep(1)

         download_link = []
         try:
             download_link = await self.tab.query_selector_all("i[class='fas fa-download']")
             print(" download_link ", len(download_link), " --- ", download_link)
             if download_link:
                 logger.info(f"Found download link with text: '{download_link}'")
         except Exception:
             pass

         if download_link:
             # Set download behavior for the browser
             await self.tab.set_download_path(self.download_path)

             list_existing_files = glob.glob(os.path.join(self.download_path,"*"+cib_code+"*.pdf"))
             file_add_name = 1
             if len(list_existing_files)>0:
                 file_add_name = max([int(i.split(os.sep)[-1].split(".pdf")[0].split(cib_code)[-1].strip().split("_")[-1]) for i in list_existing_files if i.split(os.sep)[-1].split(".pdf")[0].split(cib_code)[-1].strip().split("_")[-1]])
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             down_counter =  1
             for d_link in download_link:
                 # Click the download link
                 try:
                     await self.tab.sleep(1)
                     time.sleep(1)

                     down_xpath_link = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[2]/div[2]/br-table/div/ngx-datatable/div/div/datatable-body/datatable-selection/datatable-scroller/datatable-row-wrapper[%s]/datatable-body-row/div[2]/datatable-body-cell[7]/div/div/button/i" % str(down_counter)

                     try:
                         download_link = await self.tab.xpath(down_xpath_link, timeout=3)
                         if download_link:
                             download_link_item = download_link[0]
                             await download_link_item.click()
                             logger.info(f"Found download link with text: '{down_xpath_link}'")
                     except Exception:
                         pass

                     time.sleep(1)
                     logger.info(f"Download initiated. File should be saved to: {self.download_path}")
                     file_name_str = os.path.join(self.download_path,"Certidao-"+cib_code+".pdf")
                     if os.path.isfile(file_name_str):

                         os.rename(file_name_str, file_name_str.replace(".pdf","_"+str(file_add_name)+".pdf"))
                     file_add_name += 1
                     down_counter += 1
                 except Exception as down_error:
                     print(" dow_error : ", down_error)
                     await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             return True
         else:
             logger.warning("No download link found. The certificate may not be available.")
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

             return False

     except Exception as e:
         logger.error(f"Failed to download certificate: {str(e)}")
         return False

 async def click_return_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Voltar' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[4]/div[1]/app-voltar/button"

         try:
             return_button_element = await self.tab.xpath(button_xpath, timeout=3)
             if return_button_element:
                 return_button_element_item = return_button_element[0]
                 await return_button_element_item.click()
                 logger.info(f"Found return button with xpath: '{button_xpath}'")

                 return True
         except Exception:
             pass

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def click_new_consult_button(self):
     """Click the 'Consultar Certidão' button"""
     try:
         logger.info("Looking for 'Nova Consulta' button...")

         # Try to find the button by text
         button_xpath = "/html/body/app-root/mf-portal-layout/portal-main-layout/div/main/ng-component/ng-component/ng-component/app-resultado-consulta/div[4]/div[2]/button"

         try:
             return_button_element = await self.tab.xpath(button_xpath, timeout=3)
             # print(" return_button_element ", len(return_button_element), " --- ", return_button_element)
             if return_button_element:
                 return_button_element_item = return_button_element[0]
                 await return_button_element_item.click()
                 logger.info(f"Found New consult button with xpath: '{button_xpath}'")
                 # time.sleep(1)
                 return True
         except Exception:
             await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)

     except Exception as e:
         logger.error(f"Failed to click consultation button: {str(e)}")
         return False

 async def run(self, cib_code="", initial_date="15/08/2019", headless=False):
     """Main method to run the complete workflow"""
     # try:
     logger.info("Starting CIB certificate download process...")

     # Setup browser
     await self.setup_browser(headless=headless)

     # Navigate to CIB page
     await self.navigate_to_cib_page()

     time.sleep(1)
     # Handle cookies
     await self.handle_cookies()
     time.sleep(1)
     # Enter CIB code
     await self.enter_cib_code(cib_code)
     time.sleep(1)
     # Click consultation button
     await self.click_consult_button()
     time.sleep(1)
     # Enter initial date
     await self.enter_initial_date()
     time.sleep(1)
     # Download certificate
     success = await self.download_certificate(cib_code)

     if success:
         logger.info("CIB certificate download process completed successfully!")
     else:
         logger.warning("Process completed but certificate may not have been downloaded")

     new_consult_button = await self.click_new_consult_button()

     # print(" new_consult_button --- ", new_consult_button)
     if new_consult_button:
         logger.info("Returned to the initial page successfully!")
     else:
         logger.warning("Error in the return")
     await self.tab.save_screenshot(filename='auto', format='jpeg', full_page=True)
     return success

from xvfbwrapper import Xvfb

async def main():

 vdisplay = Xvfb()
 vdisplay.start()
 """Main entry point"""
 # Initialize the downloader
 downloader = CIBDownloader()

 # Configuration
 # '05325196'    '07447833' '45352860' '72670908' "83033467"
 CIB_CODE = "72670908"
 INITIAL_DATE = "15/08/2019"
 HEADLESS = False  # Set to True to run without GUI

 # Run the download process
 success = await downloader.run(
     cib_code=CIB_CODE,
     initial_date=INITIAL_DATE,
     headless=HEADLESS
 )
 vdisplay.stop()

if name == "main": # Run the script try: # Use nodriver's event loop uc.loop().run_until_complete(main()) except KeyboardInterrupt: print("\n⚠️ Process interrupted by user") except Exception as e: print(f"❌ Unexpected error: {str(e)}") logging.error(f"Unexpected error: {str(e)}", exc_info=True) `

As an additional info, I have been running this script for 4 days straight in my desktop using the Xvfb and it didn't have any trouble at all until now.

I try to do the same in the VPS with the same ubuntu configs and same Google Chrome version 140.0.7339.127.

It can be something related to the browser data? since at my desktop the browser has some history or different configurations maybe?

[Fedoroks2024]

Hello 👋, I would like to donate an API key from XAI to the developer. Does anyone know how to notify him and give it to him so that no one else can use it?

[thegeosman]

Any ideas in how to circle this detection problem @devblack ? We have had any success trying to avoid detection until now.