STOMP+SSL-related changes: security fixes + convenience enhancement (+ doc updates/improvements)

doc: a few fixes/improvements ad STOMP bots and *n6* feed

lib, bots, doc: only cosmetic/very minor tweaks and comments

lib, bots, pkg, doc: drop support for `stomp.py` older than 4.1.12

The affected bots are: *STOMP collector* (`StompCollectorBot` defined
in `intelmq.bots.collectors.stomp.collector`) and *STOMP output*
(`StompOutputBot` defined in `intelmq.bots.outputs.stomp.output`).

Also, in `debian/control`, the `python3-stomp` package name has been
fixed (by removing the `.py` suffix).

The changelog has been updated appropriately.

lib, bots, doc: STOMP/*n6*-related fixes/enhancements, also ad security

SSL-related changes -- regarding `intelmq.lib.mixins.StompMixin` and,
therefore, also the *STOMP collector* bot (`StompCollectorBot` defined
in `intelmq.bots.collectors.stomp.collector`) and the *STOMP output*
bot (`StompOutputBot` defined in `intelmq.bots.outputs.stomp.output`)
-- have been made:

* *Security*-focused: fixed certain security problems which were caused
  by the fact that certain versions of the `stomp.py` library we need
  to be compatible with use the `ssl` module's tools in such ways that
  suffer from certain *security weaknesses*. In particular, `stomp.py`
  in versions `>=8.0, <8.1` mistakenly creates an `SSLContext` instance
  with the `check_hostname` flag unset -- an important negative effect
  of that is that the hostname of the STOMP server is *not* checked
  during the TLS handshake (making all STOMP communication vulnerable
  to certain kinds of attacks...). Also, there are weaknesses (caused
  either by `stomp.py` or by older, yet still supported by IntelMQ,
  Python versions) of using too old versions of the TLS protocol
  (namely: 1.0 and 1.1 -- nowadays considered insecure).

* *Admin convenience*-focused: from now on, for each of the STOMP bots,
  you can set the `ssl_ca_certificate` config param to an empty string
  -- dictating that the SSL tools employed by the `stomp.py`'s machinery
  will attempt to load the system’s default CA certificates. Thanks to
  that, administrators of the given IntelMQ instance can be relieved of
  of the fuss with manual updates of the CA certificate(s) file -- *if*
  the certificate of the STOMP server can be verified using some of
  the publicly available CA certificates which are part of nearly all
  mainstream operating system distributions (this will be the case with
  the server certificate of the new variant of the *n6* Stream API, that
  is, the variant with STOMP-login-and-passcode-based authentication).

An important part of the implementation of the aforementioned changes is
a non-public class, `intelmq.lib.mixins.stomp._StompPyDedicatedSSLProxy`
-- which implements a kind of transparent proxy object that wraps the
`ssl` attribute of the `stomp.transport` module (originaly set to the
`ssl` module object), replacing some of the `ssl` module's tools with
their patched variants (note that the `ssl` module itself and all its
members are left untouched).

The parts of the IntelMQ's documentation related to those STOMP bots +
integration with *n6* (including the CERT.PL's "N6 Stomp Stream" feed
description) have been updated and improved; also, the changelog has
been updated.

bots: fix import logic in STOMP collector's module

The logic regarding importing of the `stomp.py`'s stuff has been fixed:
now the condition of the absence of the `stomp` module (and thus, of the
entire library) would not be confused with the condition of the absence
of only the `stomp.exception` module (which would mean the presence of a
version of that library lacking just the `exception` submodule).

Author: zuo

CHANGELOG.md

@@ -19,6 +19,10 @@
     if `auth_by_ssl_client_certificate` is *false*);
   - `password` (STOMP authentication passcode, default: "guest"; to be used only
     if `auth_by_ssl_client_certificate` is *false*).
+- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
+  `intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
+  to an empty string - which means that the SSL machinery used for STOMP communication
+  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
 
 ### Core
 - `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
@@ -27,7 +31,7 @@
 - `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
   which provides certain common STOMP-bot-specific operations, factored out from
   `intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
-  (PR#2408 by Jan Kaliszewski).
+  (PR#2408 and PR#2414 by Jan Kaliszewski).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -36,11 +40,16 @@
 
 ### Bots
 #### Collectors
-- `intelmq.bots.collectors.stomp.collector` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.collectors.stomp.collector` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
     fixes [#2342](https://github.com/certtools/intelmq/issues/2342).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
     never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
     did not work properly anyway).
@@ -56,27 +65,35 @@
 #### Experts
 
 #### Outputs
-- `intelmq.bots.outputs.stomp.output` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.outputs.stomp.output` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
     (`ssl_ca_cert` et consortes).
   - Add coercion of the `port` config parameter to `int`.
   - Add implementation of the `check` hook (verifying, in particular, accessibility
     of necessary file(s)).
-  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.8`).
+  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.12`).
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
 
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
 - Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
-  Stream API (PR#2408 by Jan Kaliszewski).
+  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
 
 ### Packaging
 - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
+- `debian/control`: in `Suggests` field, replace ``python3-stomp.py (>= 4.1.9)`` with
+  ``python3-stomp (>= 4.1.12)``, i.e., fix the package name by removing the `.py`
+  suffix and bump the minimum version to `4.1.12` (PR#2414 by Jan Kaliszewski).
 
 ### Tests
 

debian/control

@@ -53,7 +53,7 @@ Suggests: python3-geoip2 (>= 2.2.0),
           python3-pyasn (>= 1.5.0),
           python3-pymongo (>= 2.7.1),
           python3-sleekxmpp (>= 1.3.1),
-          python3-stomp.py (>= 4.1.9),
+          python3-stomp (>= 4.1.12),
           python3-pendulum
 Description: Solution for IT security teams for collecting and processing security feeds
  IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse

docs/user/bots.md

@@ -1176,35 +1176,49 @@ Install the `stomp.py` library from PyPI:
 pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 ```
 
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
+
+```bash
+apt install python3-stomp
+```
+
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
+
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
+
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/my.example.org/*.*.*.*`.
+
 **Parameters (also expects [feed parameters](#feed-parameters)):**
 
 **`server`**
 
-(required, string) Hostname of the STOMP server.
+(required, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" (which is default)
 
 **`port`**
 
-(optional, integer) Defaults to 61614.
+(optional, integer) STOMP server's port number (default: 61614)
 
 **`exchange`**
 
-(required, string) STOMP *destination* to subscribe to, e.g. "/exchange/my.org/*.*.*.*"
+(required, string) STOMP *destination* to subscribe to, e.g. `"/exchange/my.org/*.*.*.*"`
 
-**`username`**
-
-(optional, string) Username to use.
+**`heartbeat`**
 
-**`password`**
-
-(optional, string) Password to use.
+(optional, integer) default: 6000
 
 **`ssl_ca_certificate`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, string) Path to CA file, or empty string to load system's default CA certificates
 
 **`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+(optional, boolean) Default: true (note: false is needed for new *n6* auth)
 
 **`ssl_client_certificate`**
 
@@ -1214,6 +1228,14 @@ pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 
 (optional, string) Path to client private key to use for TLS connections.
 
+**`username`**
+
+(optional, string) Username to use.
+
+**`password`**
+
+(optional, string) Password to use.
+
 ---
 
 ### Twitter (REMOVE?) <div id="intelmq.bots.collectors.twitter.collector_twitter" />
@@ -5127,72 +5149,87 @@ This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Orient
 
 **Requirements**
 
-Install the stomp.py library, e.g. [apt install python3-stomp.py] or [pip install stomp.py].
+Install the `stomp.py` library from PyPI:
 
-You need a CA certificate, client certificate and key file from the organization / server you are connecting to. Also
-you will need a so called "exchange point".
+```bash
+pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt
+```
 
-**Parameters:**
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
 
-**`exchange`**
+```bash
+apt install python3-stomp
+```
 
-(optional, string) The exchange to push to. Defaults to `/exchange/_push`.
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
 
-**`username`**
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
 
-(optional, string) Username to use.
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/_push`.
 
-**`password`**
+**Parameters:**
 
-(optional, string) Password to use.
+**`server`**
 
-**`ssl_ca_certificate`**
+(optional, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" or "127.0.0.1" (which is default)
 
-(optional, string) Path to trusted CA certificate.
+**`port`**
 
-**`auth_by_ssl_client_certificate`**
+(optional, integer) STOMP server's port number (default: 61614)
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+**`exchange`**
+
+(optional, string) STOMP *destination* to push at, e.g. ``"/exchange/_push"`` (which is default)
 
 **`heartbeat`**
 
 (optional, integer) Defaults to 60000.
 
-**`message_hierarchical_output`**
+**`ssl_ca_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) path to CA file, or empty string to load system's default CA certificates
 
-**`message_jsondict_as_string`**
+**`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, boolean) default: true (note: false is needed for new *n6* auth)
 
-**`message_with_type`**
+**`ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) Path to client certificate to use for TLS connections.
 
-**`port`**
+**`ssl_client_certificate_key`**
 
-(optional, integer) Defaults to 61614.
+(optional, string) Path to client private key to use for TLS connections.
 
-**`server`**
+**`username`**
 
-(optional, string) Hostname of the STOMP server.
+(optional, string) STOMP *login* (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false
 
-**`single_key`**
+**`password`**
 
-(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
+(optional, string) STOMP *passcode* (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false
 
-**`ssl_ca_certificate`**
+**`message_hierarchical_output`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate`**
+**`message_jsondict_as_string`**
 
-(optional, string) Path to client certificate to use for TLS connections.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate_key`**
+**`message_with_type`**
 
-(optional, string) Path to client private key to use for TLS connections.
+(optional, boolean) Defaults to false.
+
+**`single_key`**
+
+(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
 
 ---
 

docs/user/feeds.md

@@ -719,15 +719,15 @@ parameters:
 
 ### N6 Stomp Stream
 
-N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that rate_limit does not apply for this bot as it is waiting for messages on a stream.
+N6 Collector - CERT.pl's *n6* Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
 
 **Public:** no
 
-**Revision:** 2023-09-23
+**Revision:** 2023-10-08
 
 **Documentation:** <https://n6.readthedocs.io/usage/streamapi/>
 
-**Additional Information:** Contact cert.pl to get access to the feed.
+**Additional Information:** Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new *n6* Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify "*.cert.pl" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the *new API variant*) suggested here, "n6stream-new.cert.pl", is a temporary domain; ultimately, it will be changed back to "stream.cert.pl". When it comes to the *old API variant* (turned off in November 2023!), you need to have the 'server' parameter set to the name "n6stream.cert.pl", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the *n6*'s legacy self-signed CA certificate (which is stored in file "intelmq/bots/collectors/stomp/ca.pem"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-*n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
 
 
 **Collector configuration**
@@ -736,14 +736,14 @@ N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that r
 module: intelmq.bots.collectors.stomp.collector
 parameters:
   auth_by_ssl_client_certificate: False
-  exchange: {insert your exchange point as given by CERT.pl}
+  exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}
   name: N6 Stomp Stream
-  password: {insert n6 user's API key}
+  password: {insert your *n6* API key}
   port: 61614
   provider: CERT.PL
-  server: n6stream.cert.pl
-  ssl_ca_certificate: {insert path to CA file for CERT.pl's n6}
-  username: {insert n6 user's login}
+  server: n6stream-new.cert.pl
+  ssl_ca_certificate:
+  username: {insert your *n6* login, e.g. someuser@my.example.org}
 ```
 
 **Parser configuration**

intelmq/bots/collectors/stomp/REQUIREMENTS.txt

@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: 2017 Sebastian Wagner
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-stomp.py>=4.1.8
+stomp.py>=4.1.12

intelmq/bots/collectors/stomp/collector.py

@@ -4,15 +4,19 @@
 
 # -*- coding: utf-8 -*-
 
-from intelmq.lib.bot import CollectorBot
-from intelmq.lib.mixins import StompMixin
-
 try:
     import stomp
-    import stomp.exception
 except ImportError:
     stomp = None
 else:
+    import stomp.exception
+
+from intelmq.lib.bot import CollectorBot
+from intelmq.lib.mixins import StompMixin
+
+
+if stomp is not None:
+
     class StompListener(stomp.PrintingListener):
         """
         the stomp listener gets called asynchronously for
@@ -74,17 +78,33 @@ def connect_and_subscribe(conn, logger, destination, start=False, connect_kwargs
 class StompCollectorBot(CollectorBot, StompMixin):
     """Collect data from a STOMP Interface"""
     """ main class for the STOMP protocol collector """
-    exchange: str = ''
+
+    server: str = 'n6stream.cert.pl'
     port: int = 61614
-    server: str = "n6stream.cert.pl"
-    auth_by_ssl_client_certificate: bool = True
-    username: str = 'guest'  # ignored if `auth_by_ssl_client_certificate` is true
-    password: str = 'guest'  # ignored if `auth_by_ssl_client_certificate` is true
-    ssl_ca_certificate: str = 'ca.pem'  # TODO pathlib.Path
-    ssl_client_certificate: str = 'client.pem'  # TODO pathlib.Path
-    ssl_client_certificate_key: str = 'client.key'  # TODO pathlib.Path
+    exchange: str = ''
     heartbeat: int = 6000
 
+    # Note: the `ssl_ca_certificate` configuration parameter must be set:
+    # * *either* to the server's CA certificate(s) file path,
+    # * *or* to an empty string -- dictating that the SSL tools employed
+    #   by the `stomp.py`'s machinery will attempt to load the system’s
+    #   default CA certificates.
+    # The latter, if applicable, is more convenient -- by avoiding the
+    # need to manually update the CA certificate(s) file.
+    ssl_ca_certificate: str = 'ca.pem'  # <- TODO: change to '' (+ remove "ca.pem*" legacy files)
+    # (^ TODO: could also be pathlib.Path)
+
+    auth_by_ssl_client_certificate: bool = True
+
+    # Used if `auth_by_ssl_client_certificate` is true (otherwise ignored):
+    ssl_client_certificate: str = 'client.pem'       # (cert file path)
+    ssl_client_certificate_key: str = 'client.key'   # (cert's key file path)
+    # (^ TODO: could also be pathlib.Path)
+
+    # Used if `auth_by_ssl_client_certificate` is false (otherwise ignored):
+    username: str = 'guest'   # (STOMP auth *login*)
+    password: str = 'guest'   # (STOMP auth *passcode*)
+
     _collector_empty_process: bool = True
     __conn = False  # define here so shutdown method can check for it