Should errors maybe be logged at error level?

[dogweather]

This was a tricky one for me to track down. I was getting just these logs but no actual output:

16:50:58.539 [info] Parsing https://www.oregonlegislature.gov/bills_laws/Pages/ORS.aspx...
16:51:10.270 [info] Parsing https://www.oregonlegislature.gov/bills_laws/ors/ors414.html...

...and nothing else. My logging level was at info. Since that would display any logs at a higher severity (warn, error, etc.), I couldn't understand what was happening. It took me a while to realize I had an error (!) and then another while to figure out how to see the error.

It turns out, I needed to go to debug level. It shows the errors, but also lots of debug info:

16:50:56.168 [debug] Starting requests storage worker for OrsRedirectSpider...
16:50:56.169 [debug] Started 8 workers for OrsRedirectSpider
16:50:58.539 [info] Parsing https://www.oregonlegislature.gov/bills_laws/Pages/ORS.aspx...
16:51:10.270 [info] Parsing https://www.oregonlegislature.gov/bills_laws/ors/ors414.html...
16:51:10.410 [debug] Could not parse item, error: :error, reason: :function_clause
16:51:10.410 [debug] ** (ErlangError) Erlang error: :error

So my suggestion: actual fatal errors like these that cause the Spider to crash and give up should be logged at error level.

[oltarasenko]

Yes, agree. It's a good idea to have it on the error level.