[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion

i have a complex single page web application.
i want to make a visual regression splitting the web page in different small screenshot (for example left menu, main menu, right menu, footer and navigation).

i make the baseline and screenshot are good.
i make a regression check and there is no difference.

now, if i modify something in the left menu, the test fails, but i obtain images of area that are not different **on FIREFOX, but it works on CHROME**

this is my code

```
    def test_home_mainMenu(self):
        self.driver.delete_all_cookies()
        self.driver.get('https://'+configuration.clientId+'.thron.com/#/login/'+configuration.token)
        time.sleep(15)
        self.assertScreenshot('.main-menu', 'main-menu')
        self.assertScreenshot('.apps-list-container', 'apps-list-container')
        self.assertScreenshot('.menu-right-container', 'menu-right-container')
        self.assertScreenshot('.notifications', 'notifications')
        self.assertScreenshot('.secondary-menu-contents', 'sx-menu-library')
        self.assertScreenshot('.secondary-menu-calendar', 'calendar-live')
        self.assertScreenshot('.main-primary', 'contents-area')
```

and i obtain all the correct screenshot

![image](https://cloud.githubusercontent.com/assets/2149815/15611859/970f10c0-242c-11e6-8867-671de2d469ac.png)

but now i make the test again making a change on sx-menu-library.png , closing the folder before assertScreenshot starts (this explain why there is the time.sleep(15))

so from 
![image](https://cloud.githubusercontent.com/assets/2149815/15611879/b4a2749c-242c-11e6-85b4-8d5b5b185259.png) to 
![image](https://cloud.githubusercontent.com/assets/2149815/15611887/bfa2057e-242c-11e6-9680-df72fcb90ce1.png)

test correcly fails,

but, as you can see, the screenshot created show a difference where there is no difference 

![image](https://cloud.githubusercontent.com/assets/2149815/15611948/0c987444-242d-11e6-8fe4-ddfdc94d51d7.png)

and this is the log

C:\src\automatictestingtool\NeedleVisual>nosetests homepage.py
# F
## FAIL: test_home_mainMenu (homepage.THRONVisualRegression)

Traceback (most recent call last):
  File "C:\src\automatictestingtool\NeedleVisual\homepage.py", line 15, in test_
home_mainMenu
    self.assertScreenshot('.notifications', 'notifications')
  File "c:\python27\lib\site-packages\needle\cases.py", line 131, in assertScree
nshot
    pass
  File "c:\python27\lib\contextlib.py", line 24, in **exit**
    self.gen.next()
  File "c:\python27\lib\site-packages\needle\cases.py", line 202, in compareScre
enshot
    self.engine.assertSameFiles(output_file, baseline_file, threshold)
  File "c:\python27\lib\site-packages\needle\engines\pil_engine.py", line 25, in
 assertSameFiles
    % (output_file, baseline_file, distance))
AssertionError: The new screenshot 'C:\src\automatictestingtool\NeedleVisual\scr
eenshots\notifications.png' did not match the baseline 'C:\src\automatictestingt
ool\NeedleVisual\screenshots\baseline\notifications.png' (by a distance of 198.6
1)
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1)
: thronqa-view.4me.it
requests.packages.urllib3.connectionpool: DEBUG: "POST /api/xsso/resources/ident
itymanager/login HTTP/1.1" 200 56
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session {"desiredCapabilities": {"platform": "ANY", "browserName": "firefox"
, "version": "", "marionette": false, "javascriptEnabled": true}}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/position {"y": 0
, "x": 0, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99
c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1
024, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57
4c", "height": 768}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/execute {"sessionId": "7ca39783
-4f52-4288-8afe-44a99c6b574c", "args": [], "script": "return {width: document.bo
dy.clientWidth, height: document.body.clientHeight};"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1
037, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57
4c", "height": 768}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: DELETE http://127.0.0.1:5883
5/hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/cookie {"sessionId": "7ca3978
3-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/url {"url": "https://thronqa.th
ron.com/#/login/b8171bd2-69c3-4f26-9e71-c05aa0726c47", "sessionId": "7ca39783-4f
52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".main-menu"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb
-441d9621939f}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{faf2a1af-be85-4b3d-96fb-441d9621939f}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb
-441d9621939f}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{faf2a1af-be85-4b3d-96fb-441d9621939f}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".apps-list-con
tainer"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37
-bf95d7b59c96}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{b0806653-ccfe-4755-bc37-bf95d7b59c96}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37
-bf95d7b59c96}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{b0806653-ccfe-4755-bc37-bf95d7b59c96}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".menu-right-co
ntainer"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214
-5e45bb0eb5b8}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214
-5e45bb0eb5b8}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/
hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector
", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".notifications
"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf
-080f55194a5e}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i
d": "{5d8563ea-3067-4dc7-8caf-080f55194a5e}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf
-080f55194a5e}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id":
"{5d8563ea-3067-4dc7-8caf-080f55194a5e}"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h
ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397
83-4f52-4288-8afe-44a99c6b574c"}
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8948
PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13
PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 7340
--------------------- >> end captured logging << ---------------------

---

Ran 1 test in 19.994s

FAILED (failures=1)

C:\src\automatictestingtool\NeedleVisual>

why?

thanks


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion #47

F

FAIL: test_home_mainMenu (homepage.THRONVisualRegression)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion #47

Description

F

FAIL: test_home_mainMenu (homepage.THRONVisualRegression)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions