ENH: Caching, advanced mapping and separating events for MISP Feed output bot #2509

kamil-certat · 2024-07-04T14:45:43Z

MISP Feed output bot got new features:

caching incoming messages to not re-generate on every run;
an ability to select which fields should be mapped and set additional parameters for attributes;
an ability to group messages in different MISP events, based on a field.

The bot is fully backward-compatible. By default, the previous behaviour is kept.

In addition, code related to Python 3.6 was removed and the message library was fixed not to modify the original dict instance.

This PR replaces on PR #2505.

kamil-certat · 2024-07-08T11:40:25Z

Yeah, finally green 🎉

sebix · 2024-07-08T11:51:22Z

Are you using it in production?

kamil-certat · 2024-07-08T12:55:11Z

We started using it on staging, found some pain points, and now I'll test it & hopefully promote to prod in a few days - so, not yet, but soon ;)

sebix · 2024-07-09T07:35:24Z

There's not much I can check here without reading the MISP code and docs and setting up an instance.

Maybe @Rafiot, can you have a glimpse?

aaronkaplan

minor changes requested pls.

aaronkaplan · 2024-07-09T07:49:36Z

docs/user/bots.md

 **`interval_event`**

 () The output bot creates one event per each interval, all data in this time frame is part of this event. Default "1
 hour", string.


what happens when set to 0 ?

I've looked in the code, and it looks like in such case, this would be a ValueError. However, you can use "0 hours" what should result in creating a new MISP event for every IntelMQ event.

docs/user/bots.md

aaronkaplan · 2024-07-09T08:34:55Z

docs/user/bots.md

+
+(optional, string): If set to a field name from IntelMQ event, the bot will group incoming messages
+in separated MISP events, based on the value of this field. The `interval_event` parameter acts
+for all grouping events together.


this is not 100% clear for me yet:

will it collect events until (... until when?...) and then sort and group by this viel?
I think rephrasing this slightly makes it clearer in the documentaiton.

What do you think about the current version?

will it collect events until (... until when?...) and then sort and group by this viel?

This field is responsible only for separating using the field. Collecting & holding is a different setting, that can be used together, but you don't have to. Let's have a look at the new description

intelmq/bots/outputs/misp/output_feed.py

aaronkaplan · 2024-07-09T08:41:01Z

intelmq/bots/outputs/misp/output_feed.py

+        return result

-        feed_output = self.current_event.to_feed(with_meta=False)
+    def _custom_mapping(self, obj: "MISPObject", message: dict):


Disagree it's needed here.

intelmq/bots/outputs/misp/output_feed.py

intelmq/lib/mixins/cache.py

aaronkaplan · 2024-07-09T08:47:16Z

My main request is:

give the possibility (or document it) how to add default tags to events (such as "FEED:Source=intelmq") to any event which gets generated via this feed. Adding a default tag helps in sorting out automatically added events and attributes in case something goes wrong.
make sure the users understand in the documentation how feeds work. Linking to the MISP documentation and what each parameter when including the MISP feed actually means.
For example, did you know that if you set "caching:enabled" in a MISP feed, then that means the feed events and attributes don't get added to the events (and attributes) table in MISP.
It's totally non-intuitive.
Why am I saying this? I had to figure it out the hard way myself.
basically we need a tutorial on docs.intelmq.org on how to run this properly!
It's a very cool feature and addition but you can quickly shoot yourself in the foot if you don't know how it works and what every parameter in MISP means.

kamil-certat · 2024-07-09T09:10:33Z

Thanks for the review!

re 1: It's a good idea, I'll look into it. I was concentrated on what we actually need, and I do think there are more things we can improve in the bot.
re 2: It's a good point, I will point to the MISP documentation. I also had to explore what we really can do, the PyMISP documentation wasn't really helpful :D
re 3: yeah, I agree it needs a tutorial (and I've planned it), but it's a separated thing, I won't include it here. My plan is to write a blog post once we establish the final integration internally, and then we could include it in the documentation as well. I expect to publish it around September, hopefully. There is a queue of what I'd like to write about, and a limited time ;)

kamil-certat · 2024-07-16T12:54:16Z

@aaronkaplan

Could you have a look again? I have implemented tagging as well as rewritten the documentation, added config examples, implemented validation in the check method and refactored code, including renaming methods to be more clear what they do or adding doc strings where I found it useful. I have also added descriptions to complex test cases explaining what should happen.

I do admit that the configuration of the bot is complex. I did my best to marriage flexibility and readability, but I think in the future we may eventually redesign this configuration, based on feedback. I'm also open for any suggestions.

sebix · 2024-08-27T09:42:29Z

@aaronkaplan you requested some changes to this PR and @kamil-certat implemented various changes based on your feedback. Can you please re-check?

sebix · 2025-08-29T11:41:09Z

@aaronkaplan this PR is waiting for your approval

aaronkaplan · 2025-10-23T21:08:14Z

Still need to test against a misp server

Generating MISP feed on every incoming message slows down processing. The new config option let us decide to save them in batches. Cached events are stored in a cache list in Redis. In addition, a code related to Python 3.6 was removed as we do not support this version any more.

The bot can now construct an event much more alligned to custom needs, allowing setting comments and selecting just a subset of fields to export

sebix · 2025-10-24T09:41:08Z

rebased on develop. there were quite some conflicts to fix.

sebix · 2025-10-24T09:48:20Z

now the JSON parser test fails, the output differs to the expected one.

The PR contains changes in intelmq/lib/message.py. I don't know yet if this is a rebase error or an error in the code.

With `event_separator` parameter, user can decide to create more than one MISP event in the output bot and group incomming messages based on given field.

sebix · 2025-10-24T09:54:42Z

Still need to test against a misp server

As discussed yesterday, I'll take this over

kamil-certat requested review from aaronkaplan and sebix July 4, 2024 14:45

kamil-certat force-pushed the misp_feed_mapping branch 3 times, most recently from b4f2e68 to 0e0b949 Compare July 4, 2024 14:54

kamil-certat added the component: bots label Jul 8, 2024

sebix removed their request for review July 9, 2024 07:36

aaronkaplan requested changes Jul 9, 2024

View reviewed changes

kamil-certat force-pushed the misp_feed_mapping branch from 246e181 to 05f4aef Compare July 16, 2024 12:25

kamil-certat requested a review from aaronkaplan July 25, 2024 09:21

sebix added this to the 3.5.0 milestone Aug 11, 2025

sebix assigned aaronkaplan Aug 20, 2025

sebix modified the milestones: 3.6.0, 3.5.0 Feature Release Aug 20, 2025

aaronkaplan assigned sebix Oct 23, 2025

kamil-certat added 4 commits October 24, 2025 11:32

Add documentation. Fix code compatibility

d67d6f3

Fix spelling

920a07d

ENH: Add attribute mapping

59c3014

The bot can now construct an event much more alligned to custom needs, allowing setting comments and selecting just a subset of fields to export

sebix force-pushed the misp_feed_mapping branch from 5f3f474 to 378a064 Compare October 24, 2025 09:40

sebix requested review from sebix and removed request for aaronkaplan October 24, 2025 09:42

sebix unassigned aaronkaplan Oct 24, 2025

kamil-certat added 11 commits October 24, 2025 11:53

ENH: Add support for creating separated MISP Events

b8b6061

With `event_separator` parameter, user can decide to create more than one MISP event in the output bot and group incomming messages based on given field.

FIX: Handle not existing fields with manual mapping

baef444

ENH: Add option to extend default info

cff9efe

Fix typos

f6869ab

ENH: add support for tagging

bd7e0d1

Fix generating on restart

820bdec

ENH: Add tagging, check, and improved docs

d05aceb

DOC: Update documentation about CacheMixin

26f161f

Adjust to pycodestyle

93790df

Fix typo

1a980e0

Clean up imports in tests

6577a90

sebix force-pushed the misp_feed_mapping branch from 378a064 to 6577a90 Compare October 24, 2025 09:54

ENH: Caching, advanced mapping and separating events for MISP Feed output bot #2509

Are you sure you want to change the base?

ENH: Caching, advanced mapping and separating events for MISP Feed output bot #2509

Uh oh!

Conversation

kamil-certat commented Jul 4, 2024 • edited by sebix Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kamil-certat commented Jul 8, 2024

Uh oh!

sebix commented Jul 8, 2024

Uh oh!

kamil-certat commented Jul 8, 2024

Uh oh!

sebix commented Jul 9, 2024

Uh oh!

aaronkaplan left a comment

Choose a reason for hiding this comment

Uh oh!

aaronkaplan Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

kamil-certat Jul 16, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aaronkaplan Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

kamil-certat Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronkaplan Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

kamil-certat Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aaronkaplan commented Jul 9, 2024

Uh oh!

kamil-certat commented Jul 9, 2024

Uh oh!

kamil-certat commented Jul 16, 2024

Uh oh!

sebix commented Aug 27, 2024

Uh oh!

sebix commented Aug 29, 2025

Uh oh!

aaronkaplan commented Oct 23, 2025

Uh oh!

sebix commented Oct 24, 2025

Uh oh!

sebix commented Oct 24, 2025

Uh oh!

sebix commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kamil-certat commented Jul 4, 2024 •

edited by sebix

Loading