add actions support

Author: Germey

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Gerapy Pyppeteer Changelog
 
+## 0.2.2 (2021-09-07)
+
+### Features
+
+- add support for executing Python based functions
+- add support for returning script result
+
 ## 0.1.2 (2021-06-20)
 
 ### Buf Fixes & Features

README.md

@@ -36,7 +36,7 @@ web page which you configured the request as PyppeteerRequest.
 
 GerapyPyppeteer provides some optional settings.
 
-### Concurrency 
+### Concurrency
 
 You can directly use Scrapy's setting to set Concurrency of Pyppeteer,
 for example:
@@ -47,7 +47,7 @@ CONCURRENT_REQUESTS = 3
 
 ### Pretend as Real Browser
 
-Some website will detect WebDriver or Headless, GerapyPyppeteer can 
+Some website will detect WebDriver or Headless, GerapyPyppeteer can
 pretend Chromium by inject scripts. This is enabled by default.
 
 You can close it if website does not detect WebDriver to speed up:
@@ -56,15 +56,15 @@ You can close it if website does not detect WebDriver to speed up:
 GERAPY_PYPPETEER_PRETEND = False
 ```
 
-Also you can use `pretend` attribute in `PyppeteerRequest` to overwrite this 
+Also you can use `pretend` attribute in `PyppeteerRequest` to overwrite this
 configuration.
 
 ### Logging Level
 
 By default, Pyppeteer will log all the debug messages, so GerapyPyppeteer
 configured the logging level of Pyppeteer to WARNING.
 
-If you want to see more logs from Pyppeteer, you can change the this setting: 
+If you want to see more logs from Pyppeteer, you can change the this setting:
 
 ```python
 import logging
@@ -82,11 +82,11 @@ GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT = 30
 
 ### Headless
 
-By default, Pyppeteer is running in `Headless` mode, you can also 
+By default, Pyppeteer is running in `Headless` mode, you can also
 change it to `False` as you need, default is `True`:
 
 ```python
-GERAPY_PYPPETEER_HEADLESS = False 
+GERAPY_PYPPETEER_HEADLESS = False
 ```
 
 ### Window Size
@@ -137,19 +137,19 @@ GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES = ['stylesheet', 'script']
 
 All of the optional resource type list:
 
-* document: the Original HTML document
-* stylesheet: CSS files
-* script: JavaScript files
-* image: Images
-* media: Media files such as audios or videos
-* font: Fonts files
-* texttrack: Text Track files
-* xhr: Ajax Requests
-* fetch: Fetch Requests
-* eventsource: Event Source
-* websocket: Websocket
-* manifest: Manifest files
-* other: Other files
+- document: the Original HTML document
+- stylesheet: CSS files
+- script: JavaScript files
+- image: Images
+- media: Media files such as audios or videos
+- font: Fonts files
+- texttrack: Text Track files
+- xhr: Ajax Requests
+- fetch: Fetch Requests
+- eventsource: Event Source
+- websocket: Websocket
+- manifest: Manifest files
+- other: Other files
 
 ### Screenshot
 
@@ -158,7 +158,7 @@ You can get screenshot of loaded page, you can pass `screenshot` args to `Pyppet
 - `type` (str): Specify screenshot type, can be either `jpeg` or `png`. Defaults to `png`.
 - `quality` (int): The quality of the image, between 0-100. Not applicable to `png` image.
 - `fullPage` (bool): When true, take a screenshot of the full scrollable page. Defaults to `False`.
-- `clip`  (dict): An object which specifies clipping region of the page. This option should have the following fields:
+- `clip` (dict): An object which specifies clipping region of the page. This option should have the following fields:
   - `x` (int): x-coordinate of top-left corner of clip area.
   - `y` (int): y-coordinate of top-left corner of clip area.
   - `width` (int): width of clipping area.
@@ -200,41 +200,69 @@ GERAPY_PYPPETEER_SCREENSHOT = {
 
 `PyppeteerRequest` provide args which can override global settings above.
 
-* url: request url
-* callback: callback
-* one of "load", "domcontentloaded", "networkidle0", "networkidle2".
-        see https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goto, default is `domcontentloaded`
-* wait_for: wait for some element to load, also supports dict
-* script: script to execute
-* proxy: use proxy for this time, like `http://x.x.x.x:x`
-* sleep: time to sleep after loaded, override `GERAPY_PYPPETEER_SLEEP`
-* timeout: load timeout, override `GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT`
-* ignore_resource_types: ignored resource types, override `GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES`
-* pretend: pretend as normal browser, override `GERAPY_PYPPETEER_PRETEND`
-* screenshot: ignored resource types, see
-        https://miyakogi.github.io/pyppeteer/_modules/pyppeteer/page.html#Page.screenshot,
-        override `GERAPY_PYPPETEER_SCREENSHOT`
+- url: request url
+- callback: callback
+- one of "load", "domcontentloaded", "networkidle0", "networkidle2".
+  see https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goto, default is `domcontentloaded`
+- wait_for: wait for some element to load, also supports dict
+- script: script to execute
+- proxy: use proxy for this time, like `http://x.x.x.x:x`
+- sleep: time to sleep after loaded, override `GERAPY_PYPPETEER_SLEEP`
+- timeout: load timeout, override `GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT`
+- ignore_resource_types: ignored resource types, override `GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES`
+- pretend: pretend as normal browser, override `GERAPY_PYPPETEER_PRETEND`
+- screenshot: ignored resource types, see
+  https://miyakogi.github.io/pyppeteer/_modules/pyppeteer/page.html#Page.screenshot,
+  override `GERAPY_PYPPETEER_SCREENSHOT`
 
 For example, you can configure PyppeteerRequest as:
 
 ```python
 from gerapy_pyppeteer import PyppeteerRequest
 
 def parse(self, response):
-    yield PyppeteerRequest(url, 
+    yield PyppeteerRequest(url,
         callback=self.parse_detail,
         wait_until='domcontentloaded',
         wait_for='title',
-        script='() => { console.log(document) }',
+        script='() => { return {name: "Germey"} }',
         sleep=2)
 ```
 
 Then Pyppeteer will:
-* wait for document to load
-* wait for title to load
-* execute `console.log(document)` script
-* sleep for 2s
-* return the rendered web page content
+
+- wait for document to load
+- wait for title to load
+- execute `console.log(document)` script
+- sleep for 2s
+- return the rendered web page content, get from `response.meta['screenshot']`
+- return the script executed result, get from `response.meta['script_result']`
+
+For waiting mechanism controlled by JavaScript, you can use await in `script`, for example:
+
+```python
+js = '''async () => {
+    await new Promise(resolve => setTimeout(resolve, 10000));
+    return {
+        'name': 'Germey'
+    }
+}
+'''
+yield PyppeteerRequest(url, callback=self.parse, script=js)
+```
+
+Then you can get the script result from `response.meta['script_result']`, result is `{'name': 'Germey'}`.
+
+If you think the JavaScript is wired to write, you can use actions argument to define a function to execute `Python` based functions, for example:
+
+```python
+async def execute_actions(page: Page):
+    await page.evaluate('() => { document.title = "Hello World"; }')
+    return 1
+yield PyppeteerRequest(url, callback=self.parse, actions=execute_actions)
+```
+
+Then you can get the actions result from `response.meta['actions_result']`, result is `1`.
 
 ## Example
 
@@ -366,4 +394,3 @@ chromiumExecutable = {
 ```
 
 You can find your own operating system, modify your chrome or chrome executable path.
-

example/example/settings.py

@@ -17,7 +17,7 @@
 
 RETRY_HTTP_CODES = [403, 500, 502, 503, 504]
 
-GERAPY_PYPPETEER_HEADLESS = True
+GERAPY_PYPPETEER_HEADLESS = False
 
 LOG_LEVEL = 'DEBUG'
 

example/example/spiders/book.py

@@ -4,10 +4,24 @@
 from example.items import BookItem
 from gerapy_pyppeteer import PyppeteerRequest
 import logging
+from pyppeteer.page import Page
 
 logger = logging.getLogger(__name__)
 
 
+js = '''async () => {
+    await new Promise(resolve => setTimeout(resolve, 10000));
+    return {
+        'name': 'Germey'
+    }
+}'''
+
+
+async def execute_action(page: Page):
+    await page.evaluate('() => { document.title = "Hello World"; }')
+    return 1
+
+
 class BookSpider(scrapy.Spider):
     name = 'book'
     allowed_domains = ['spa5.scrape.center']
@@ -20,14 +34,15 @@ def start_requests(self):
         """
         start_url = f'{self.base_url}/page/1'
         logger.info('crawling %s', start_url)
-        yield PyppeteerRequest(start_url, callback=self.parse_index, wait_for='.item .name')
+        yield PyppeteerRequest(start_url, callback=self.parse_index, actions=execute_action, wait_for='.item .name', script=js)
 
     def parse_index(self, response):
         """
         extract books and get next page
         :param response:
         :return:
         """
+        logger.debug('response meta %s', response.meta)
         items = response.css('.item')
         for item in items:
             href = item.css('.top a::attr(href)').extract_first()

gerapy_pyppeteer/__version__.py

@@ -1,3 +1,3 @@
-VERSION = (0, 1, '2')
+VERSION = (0, 2, '2rc1')
 
 version = __version__ = '.'.join(map(str, VERSION))

gerapy_pyppeteer/downloadermiddlewares.py

@@ -312,11 +312,19 @@ async def _handle_interception(pu_request):
                 await browser.close()
                 return self._retry(request, 504, spider)
 
+        _actions_result = None
+        # evaluate actions
+        if pyppeteer_meta.get('actions'):
+            _actions = pyppeteer_meta.get('actions')
+            logger.debug('evaluating %s', _actions)
+            _actions_result = await _actions(page)
+
+        _script_result = None
         # evaluate script
         if pyppeteer_meta.get('script'):
             _script = pyppeteer_meta.get('script')
             logger.debug('evaluating %s', _script)
-            await page.evaluate(_script)
+            _script_result = await page.evaluate(_script)
 
         # sleep
         _sleep = self.sleep
@@ -365,6 +373,10 @@ async def _handle_interception(pu_request):
             encoding='utf-8',
             request=request
         )
+        if _script_result:
+            response.meta['script_result'] = _script_result
+        if _actions_result:
+            response.meta['actions_result'] = _actions_result
         if screenshot:
             response.meta['screenshot'] = screenshot
         return response

gerapy_pyppeteer/request.py

@@ -6,8 +6,8 @@ class PyppeteerRequest(Request):
     """
     Scrapy ``Request`` subclass providing additional arguments
     """
-    
-    def __init__(self, url, callback=None, wait_until=None, wait_for=None, script=None, proxy=None,
+
+    def __init__(self, url, callback=None, wait_until=None, wait_for=None, script=None, actions=None, proxy=None,
                  sleep=None, timeout=None, ignore_resource_types=None, pretend=None, screenshot=None, meta=None, *args,
                  **kwargs):
         """
@@ -17,6 +17,7 @@ def __init__(self, url, callback=None, wait_until=None, wait_for=None, script=No
                 see https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goto, default is `domcontentloaded`
         :param wait_for: wait for some element to load, also supports dict
         :param script: script to execute
+        :param actions: actions defined for execution of Page object
         :param proxy: use proxy for this time, like `http://x.x.x.x:x`
         :param sleep: time to sleep after loaded, override `GERAPY_PYPPETEER_SLEEP`
         :param timeout: load timeout, override `GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT`
@@ -31,29 +32,38 @@ def __init__(self, url, callback=None, wait_until=None, wait_for=None, script=No
         # use meta info to save args
         meta = copy.deepcopy(meta) or {}
         pyppeteer_meta = meta.get('pyppeteer') or {}
-        
+
         self.wait_until = pyppeteer_meta.get('wait_until') if pyppeteer_meta.get(
             'wait_until') is not None else (wait_until or 'domcontentloaded')
-        self.wait_for = pyppeteer_meta.get('wait_for') if pyppeteer_meta.get('wait_for') is not None else wait_for
-        self.script = pyppeteer_meta.get('script') if pyppeteer_meta.get('script') is not None else script
-        self.sleep = pyppeteer_meta.get('sleep') if pyppeteer_meta.get('sleep') is not None else sleep
-        self.proxy = pyppeteer_meta.get('proxy') if pyppeteer_meta.get('proxy') is not None else proxy
-        self.pretend = pyppeteer_meta.get('pretend') if pyppeteer_meta.get('pretend') is not None else pretend
-        self.timeout = pyppeteer_meta.get('timeout') if pyppeteer_meta.get('timeout') is not None else timeout
+        self.wait_for = pyppeteer_meta.get('wait_for') if pyppeteer_meta.get(
+            'wait_for') is not None else wait_for
+        self.script = pyppeteer_meta.get('script') if pyppeteer_meta.get(
+            'script') is not None else script
+        self.actions = pyppeteer_meta.get('actions') if pyppeteer_meta.get(
+            'actions') is not None else actions
+        self.sleep = pyppeteer_meta.get('sleep') if pyppeteer_meta.get(
+            'sleep') is not None else sleep
+        self.proxy = pyppeteer_meta.get('proxy') if pyppeteer_meta.get(
+            'proxy') is not None else proxy
+        self.pretend = pyppeteer_meta.get('pretend') if pyppeteer_meta.get(
+            'pretend') is not None else pretend
+        self.timeout = pyppeteer_meta.get('timeout') if pyppeteer_meta.get(
+            'timeout') is not None else timeout
         self.ignore_resource_types = pyppeteer_meta.get('ignore_resource_types') if pyppeteer_meta.get(
             'ignore_resource_types') is not None else ignore_resource_types
         self.screenshot = pyppeteer_meta.get('screenshot') if pyppeteer_meta.get(
             'screenshot') is not None else screenshot
-        
+
         pyppeteer_meta = meta.setdefault('pyppeteer', {})
         pyppeteer_meta['wait_until'] = self.wait_until
         pyppeteer_meta['wait_for'] = self.wait_for
         pyppeteer_meta['script'] = self.script
+        pyppeteer_meta['actions'] = self.actions
         pyppeteer_meta['sleep'] = self.sleep
         pyppeteer_meta['proxy'] = self.proxy
         pyppeteer_meta['pretend'] = self.pretend
         pyppeteer_meta['timeout'] = self.timeout
         pyppeteer_meta['screenshot'] = self.screenshot
         pyppeteer_meta['ignore_resource_types'] = self.ignore_resource_types
-        
+
         super().__init__(url, callback, meta=meta, *args, **kwargs)