security.txt expert: fixes, updates to current standards

Author: sebix

CHANGELOG.md

@@ -27,6 +27,8 @@
   - Fix to avoid schema download if not configured #2530.
 
 #### Experts
+- `intelmq.bots.experts.securitytxt`:
+  - Added new bot (PR#2538 by Frank Westers and Sebastian Wagner)
 
 #### Outputs
 

docs/user/bots.md

@@ -3528,7 +3528,10 @@ true.
 
 ### SecurityTXT <div id="intelmq.bots.experts.securitytxt.expert" />
 
-SecurityTXT is an initiative to standardize how websites publish their abuse contact information. Currently it is a `draft RFC <https://datatracker.ietf.org/doc/html/draft-foudil-securitytxt-12>`_. See this draft RFC for more information on security.txt. This bot automatically looks for security.txt files on a URL or IP, retrieves the primary contact information out of it and adds this to the event.
+SecurityTXT is an initiative to standardize how websites publish their abuse contact information.
+It is standardized in [RFC 9116 "A File Format to Aid in Security Vulnerability Disclosure"](https://datatracker.ietf.org/doc/rfc9116/).
+Refer to the linked document RFC for more information on `security.txt`.
+This bot looks for `security.txt` files on a URL or IP, retrieves the primary contact information out of it and adds this to the event.
 
 **Requirements**
 
@@ -3540,31 +3543,35 @@ pip3 install -r intelmq/bots/experts/securitytxt/REQUIREMENTS.txt
 
 **Module:** `intelmq.bots.experts.securitytxt.expert`
 
-**Parameters (also expects [cache parameters](#cache-parameters)):**
+**Parameters**
 
 **`url_field`**
 
-The field in the event that contains the URL/IP on which to look for the the security.txt file.
+The field in the event that contains the URL/IP on which to look for the the security.txt file. Default: `source.reverse_dns`
 
 **`contact_field`**
 
-The field in the event in which to put the found contact details
+The field in the event in which to put the found contact details. Default: `source.abuse_contact`
 
-**`only_email_address`**
+**`only_email_address`** (bool)
 
 Contact details can be web URLs or email addresses. When this value is set to True, it only selects email addresses as contact information.
+Default: `true`
 
-**`overwrite`**
+**`overwrite`** (bool)
 
-Boolean indicating whether to override existing data in contact_field
+Boolean indicating whether to override existing data in contact_field.
+Default: `true`
 
-**`check_expired`**
+**`check_expired`** (bool)
 
-Boolean indicating whether to check if the security.txt has expired according to its own expiry date
+Boolean indicating whether to check if the security.txt has expired according to its own expiry date.
+Default: `false`
 
-**`check_canonical`**
+**`check_canonical`** (bool)
 
 Boolean indicating whether to check if the url is contained in the list of canonical urls.
+Default: `false`
 
 
 ---

intelmq/bots/experts/securitytxt/REQUIREMENTS.txt

@@ -1 +1,4 @@
+# SPDX-FileCopyrightText: 2022 Frank Westers, 2024 Institute for Common Good Technology
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
 wellknown-securitytxt

intelmq/bots/experts/securitytxt/expert.py

@@ -1,9 +1,18 @@
+# SPDX-FileCopyrightText: 2022 Frank Westers, 2024 Institute for Common Good Technology
+#
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
 from typing import Optional
 
 import requests
-from securitytxt import SecurityTXT
 
 from intelmq.lib.bot import ExpertBot
+from intelmq.lib.exceptions import MissingDependencyError
+
+try:
+    from securitytxt import SecurityTXT
+except (ImportError, ModuleNotFoundError):
+    SecurityTXT = None
 
 
 class SecurityTXTExpertBot(ExpertBot):
@@ -27,8 +36,8 @@ class SecurityTXTExpertBot(ExpertBot):
     check_canonical: bool = False
 
     def init(self):
-        if not self.url_field or not self.contact_field:
-            raise AttributeError("Not all required fields are set.")
+        if SecurityTXT is None:
+            raise MissingDependencyError('wellknown-securitytxt')
 
     def process(self):
         event = self.receive_message()
@@ -38,9 +47,9 @@ def process(self):
             primary_contact = self.get_primary_contact(event.get(self.url_field))
             event.add(self.contact_field, primary_contact, overwrite=self.overwrite)
         except NotMeetsRequirementsError as e:
-            self.logger.debug(str(e) + " Skipping event.")
+            self.logger.debug("Skipping event (%s).", e)
         except ContactNotFoundError as e:
-            self.logger.debug(f"No contact found. {str(e)} Continue.")
+            self.logger.debug("No contact found: %s Continue.", e)
 
         self.send_message(event)
         self.acknowledge_message()
@@ -101,4 +110,4 @@ class ContactNotFoundError(Exception):
     pass
 
 
-BOT = SecurityTXTExpertBot
+BOT = SecurityTXTExpertBot

intelmq/tests/bots/experts/securitytxt/REQUIREMENTS.txt

@@ -1,3 +1,5 @@
+# SPDX-FileCopyrightText: 2022 Frank Westers
+#
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
 # -*- coding: utf-8 -*-
@@ -30,6 +32,7 @@
                                    "source.fqdn": "test.local"}
 
 @requests_mock.Mocker()
+@test.skip_exotic()
 class TestSecurityTXTExpertBot(test.BotTestCase, unittest.TestCase):
     """
     A TestCase for the SecurityTXT Expert Bot