About creating a custom Docker image

[baphael]

Hello !
I was wondering whether EA2 was compatible with Python 3.10 or not.
I've built my own EA2 docker image based on python:3-alpine and I'm facing the following error message when running it :

INFO:elastalert:Queried rule test from 2022-02-08 18:30 +0000 to 2022-02-08 18:31 +0000: 191 / 191 hits
ERROR:elastalert:Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/elastalert/elastalert.py", line 1298, in handle_rule_execution
    num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
  File "/opt/venv/lib/python3.10/site-packages/elastalert/elastalert.py", line 926, in run_rule
    if not self.run_query(rule, rule['starttime'], endtime):
  File "/opt/venv/lib/python3.10/site-packages/elastalert/elastalert.py", line 652, in run_query
    data = self.get_hits(rule, start, end, index, scroll)
  File "/opt/venv/lib/python3.10/site-packages/elastalert/elastalert.py", line 440, in get_hits
    hits = self.process_hits(rule, hits)
  File "/opt/venv/lib/python3.10/site-packages/elastalert/elastalert.py", line 340, in process_hits
    set_es_key(hit['_source'], rule['timestamp_field'], rule['ts_to_dt'](ts))
  File "/opt/venv/lib/python3.10/site-packages/elastalert/util.py", line 146, in ts_to_dt
    dt = dateutil.parser.parse(timestamp)
  File "/opt/venv/lib/python3.10/site-packages/dateutil/parser.py", line 1182, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/dateutil/parser.py", line 587, in parse
    if (isinstance(tzinfos, collections.Callable) or
AttributeError: module 'collections' has no attribute 'Callable'

ERROR:elastalert:Uncaught exception running rule test: module 'collections' has no attribute 'Callable'

I think it might have something to do with this (from Python docs) :

Remove deprecated aliases to Collections Abstract Base Classes from the collections module. (Contributed by Victor Stinner in bpo-37324.)

Can someone confirm ?

In the meantime, I assume we should stay on Python <= 3.9...

[wajika]

I had a similar problem 703 and I'm not sure if it's a compatibility issue .

Regarding the python version, I saw similar instructions in the doc, it is definitely said that elastalert2 supports python 3.10 version
requirements

[nsano-rururu]

I installed it on python 3.10 and ran it, but there was no error. How can I reproduce it?

[nsano-rururu]

@raph-berteaud

• Are you installing in a python virtual environment?
• Execute the following command.

pip list | grep python-dateutil

[nsano-rururu]

If it is not the following version, please upgrade python-dateutil to 2.8.2.

python-dateutil         2.8.2

[baphael]

@nsano-rururu Thanks for your answer !

• EA2 and dependencies are installed in a virtualenv
• python-dateutil is installed, but version 2.6.1...

Here is a partial output of docker image build process :

[...]
Collecting python-dateutil<2.7.0,>=2.6.0
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194 kB)
[...]

Below, the output of cat /path_to_venv/lib/python3.10/site-packages/elastalert2-2.0.3.dist-info/METADATA :

Metadata-Version: 2.1
Name: elastalert2
Version: 2.0.3
Summary: Runs custom filters on Elasticsearch and alerts on matches
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: Apache 2.0
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.9
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Dist: apscheduler (<4.0,>=3.3.0)
Requires-Dist: aws-requests-auth (>=0.3.0)
Requires-Dist: sortedcontainers (>=2.2.2)
Requires-Dist: boto3 (>=1.4.4)
Requires-Dist: configparser (>=3.5.0)
Requires-Dist: croniter (>=0.3.16)
Requires-Dist: elasticsearch (<8.0.0,>=7.0.0)
Requires-Dist: envparse (>=0.2.0)
Requires-Dist: exotel (>=0.1.3)
Requires-Dist: jira (>=2.0.0)
Requires-Dist: Jinja2 (==2.11.3)
Requires-Dist: jsonschema (>=3.0.2)
Requires-Dist: mock (>=2.0.0)
Requires-Dist: prison (>=0.1.2)
Requires-Dist: prometheus-client (>=0.10.1)
Requires-Dist: py-zabbix (>=1.1.3)
Requires-Dist: PyStaticConfiguration (>=0.10.3)
Requires-Dist: python-dateutil (<2.7.0,>=2.6.0)
Requires-Dist: PyYAML (>=5.1)
Requires-Dist: requests (>=2.10.0)
Requires-Dist: stomp.py (>=4.1.17)
Requires-Dist: texttable (>=0.8.8)
Requires-Dist: twilio (<6.1,>=6.0.0)
Requires-Dist: cffi (>=1.11.5)
Requires-Dist: tzlocal (<3.0)

UNKNOWN

Is there not something weird here ?

---

A few more details :

• EA2 is installed through pip (python -m pip install elastalert2)
• Other dependencies installed through pip are :
  • pip (upgrade)
  • wheel
  • elasticsearch==7.16.2
  • mysqlclient

Thanks again. 👍

[baphael]

I hope you're kidding !

I understand that you cannot spend time answering questions about a sort of custom setup... That said, either the translation is quite poor or you are disrespectful. Anyway, I don't see how I am to blame here.

Have a nice day and sorry for "wasting your personal time" !

[nsano-rururu]

Yelp/elastalert#2593 (comment)

Also note: I also added a pin for elasticsearch==7.0.0, because apparently 7.1.0 will NOT work with ES < 6.6 due to it not supported _source_include(s?). 7.0.0 does. Tests won't pass otherwise.

Fix issue caused by 7.x breaking change (_source_include/_source_exclude)
elastic/elasticsearch-py#1019

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#source-include-exclude-params-removed

Source filtering url parameters _source_include and _source_exclude have been removed
The deprecated in 6.x url parameters are now removed. Use _source_includes and _source_excludes instead.

ES Version revert to 7.0.0
#90

[nsano-rururu]

doc_type is no longer supported in ES > 7.x , however use_count_query and use_terms_query still check for it #2424
Yelp/elastalert#2424

[jertel]

PR #102 has some background on why the official Docker image is no longer using Alpine.

I've updated the Project Guidelines (#11) with a brief paragraph to help prevent confusion or frustration between project maintainers and those seeking assistance. Specifically, the following:

If you suspect a bug in the project, we ask that you provide a reproducible test case via the provided Docker image, as it's difficult and time-consuming to distinguish bugs from user environmental issues. If you are attempting to install ElastAlert 2 on an existing environment, or attempting to build your own custom Docker image, you are welcome to ask others for assistance. If there is no response to your discussion topic, it could mean that the project maintainers do not have additional time to devote to that form of assistance.

[nsano-rururu]

Support for custom docker images requires that you include a Dockerfile.