ENH: Caching, advanced mapping and separating events for MISP Feed output bot

[kamil-certat]

MISP Feed output bot got new features:

• caching incoming messages to not re-generate on every run;
• an ability to select which fields should be mapped and set additional parameters for attributes;
• an ability to group messages in different MISP events, based on a field.

The bot is fully backward-compatible. By default, the previous behaviour is kept.

In addition, code related to Python 3.6 was removed and the message library was fixed not to modify the original dict instance.

This PR replaces on PR #2505.

[kamil-certat]

Yeah, finally green 🎉

[sebix]

Are you using it in production?

[kamil-certat]

We started using it on staging, found some pain points, and now I'll test it & hopefully promote to prod in a few days - so, not yet, but soon ;)

[sebix]

There's not much I can check here without reading the MISP code and docs and setting up an instance.

Maybe @Rafiot, can you have a glimpse?

[aaronkaplan]

minor changes requested pls.

[aaronkaplan]

this is not 100% clear for me yet:

will it collect events until (... until when?...) and then sort and group by this viel?
I think rephrasing this slightly makes it clearer in the documentaiton.

[kamil-certat]

What do you think about the current version?

will it collect events until (... until when?...) and then sort and group by this viel?

This field is responsible only for separating using the field. Collecting & holding is a different setting, that can be used together, but you don't have to. Let's have a look at the new description

[aaronkaplan]

same

[kamil-certat]

Disagree it's needed here.

[aaronkaplan]

My main request is:

1. give the possibility (or document it) how to add default tags to events (such as "FEED:Source=intelmq") to any event which gets generated via this feed. Adding a default tag helps in sorting out automatically added events and attributes in case something goes wrong.
2. make sure the users understand in the documentation how feeds work. Linking to the MISP documentation and what each parameter when including the MISP feed actually means.
   For example, did you know that if you set "caching:enabled" in a MISP feed, then that means the feed events and attributes don't get added to the events (and attributes) table in MISP.
   It's totally non-intuitive.
   Why am I saying this? I had to figure it out the hard way myself.
3. basically we need a tutorial on docs.intelmq.org on how to run this properly!
   It's a very cool feature and addition but you can quickly shoot yourself in the foot if you don't know how it works and what every parameter in MISP means.

[kamil-certat]

Thanks for the review!

re 1: It's a good idea, I'll look into it. I was concentrated on what we actually need, and I do think there are more things we can improve in the bot.
re 2: It's a good point, I will point to the MISP documentation. I also had to explore what we really can do, the PyMISP documentation wasn't really helpful :D
re 3: yeah, I agree it needs a tutorial (and I've planned it), but it's a separated thing, I won't include it here. My plan is to write a blog post once we establish the final integration internally, and then we could include it in the documentation as well. I expect to publish it around September, hopefully. There is a queue of what I'd like to write about, and a limited time ;)

[kamil-certat]

@aaronkaplan

Could you have a look again? I have implemented tagging as well as rewritten the documentation, added config examples, implemented validation in the check method and refactored code, including renaming methods to be more clear what they do or adding doc strings where I found it useful. I have also added descriptions to complex test cases explaining what should happen.

I do admit that the configuration of the bot is complex. I did my best to marriage flexibility and readability, but I think in the future we may eventually redesign this configuration, based on feedback. I'm also open for any suggestions.

[sebix]

@aaronkaplan you requested some changes to this PR and @kamil-certat implemented various changes based on your feedback. Can you please re-check?

[sebix]

@aaronkaplan this PR is waiting for your approval

[aaronkaplan]

Still need to test against a misp server

[sebix]

Still need to test against a misp server

As discussed yesterday, I'll take this over

[sebix]

Found problems in the docs and in the changes of the cache mixin

[kamil-certat]

I'm working on a few improvements / additional requirements right now, a new code should come today or tomorrow. I'll then address comments

[sebix]

Documentation for the new parameter flat_events is missing

[kamil-certat]

I'm aware of that, I'll adjust documentation tomorrow

[kamil-certat]

I also have to add matching unit tests ;)

requested changes were implemented

[kamil-certat]

BTW, all methods were documented in bot-development 🤔

[sebix]

BTW, all methods were documented in bot-development 🤔

Oh, sorry. I don't know where I was looking.