Notify flag in the DHO

[sykaeh]

I would like to add a "notify" flag to the data harmonization ontology (DHO). This field specifies whether the responsible parties should be notified about this event. Our use case is that an expert then decides based on the attributes of the event (classification.type, classification.identifier, ip, time.source, etc.) whether people should be notified about this event and sets the flag appropriately. This can then be used to filter events when sending notifications/uploading to a database etc. Is anyone else interested in such a flag?

[aaronkaplan]

On 27 Oct 2016, at 18:08, Sybil Ehrensberger notifications@github.com wrote:

I would like to add a "notify" flag to the data harmonization ontology (DHO). This field specifies whether the responsible parties should be notified about this event.

okay. So the notify flag actually already exists in cert.at's version.
We were not sure if it is generally relevant. But it seems to be. So how about putting that "upstream" to the main certtools/intelmq repo?

Our use case is that an expert then decides based on the attributes of the event (classification.type, classification.identifier, ip, time.source, etc.) whether people should be notified about this event and sets the flag appropriately. This can then be used to filter events when sending notifications/uploading to a database etc. Is anyone else interested in such a flag?

Excellent.
Yes, we have it already.

Anyone else? If so, I'll merge that upstream.

Best,
a.

[bernhard-herzog]

We're going to need something similar, but more refined. A simple boolean flag would not enough. We have to be able to decide and record in the event who to notify in what way. Our current plans basically require some JSON structure for this.

[aaronkaplan]

---

Mobile

On 27.10.2016, at 20:36, Bernhard Herzog notifications@github.com wrote:

We're going to need something similar, but more refined. A simple boolean flag would not enough. We have to be able to decide and record in the event who to notify in what way. Our current plans basically require some JSON structure for this.

So am i hearing that you need some
Kind of variable which tracks some state of an event? ( and that notify Boolean would not be enough)?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[dmth]

So am i hearing that you need some Kind of variable which tracks some state of an event? ( and that notify Boolean would not be enough)?

I'll try to elaborate a little bit: Currently we are "mis-using" the extras field in order to store information about contacts and how they should be informed. This information can contain multiple contacts with a different set of rules. We'd appreciate a field which is intended to contain such information. But this field cannot be a boolean, because we support the notification of multiple contacts.

notify Is a good name, but maybe a little bit to specific for a use-case were this field might contain specific processing information, like provenance of data.

Such a processing-field might contain something like this (fictional structure below):

"processing_info": [
  "dns_lookup_expert": {
    "used_nameserver": "ns1.example.com",
    "lookup_timestamp": 4711
  },
  "flow-monitor": {
    "timestamp": "4712"
  },
  "notifications": [{
     "who": "mail@example.com"
     "how": "e-mail"
     "when": "immediately"
    }, {
     "who": "mail@test.com"
     "how": "e-mail"
     "when": "twice-a-day"
    }
     "who": "mail@test.com"
     "how": "xmpp"
    }
 }
]

[sebix]

I'll try to elaborate a little bit: Currently we are "mis-using" the extras field

You can always extend the harmonization if it makes sense for your setup and there's no usecase for sharing this with other instances.

Concerning your proposal:
I think both parts, information from lookup bots and the blown up structure for storing abuse contact data is opposing key principles of intelmq.

How do you mark events as "don't notify"?

Assigning this to @aaronkaplan as it affects the harmonization

[dmth]

You can always extend the harmonization if it makes sense for your setup and there's no usecase for sharing this with other instances.

We doubt that there is no usecase for other instances, that's why we are proposing to create a field which is dedicated for this processing data.

Concerning your proposal:
I think both parts, information from lookup bots and the blown up structure for storing abuse contact data is opposing key principles of intelmq.

Although destination.abuse_contact and source.abuse_contact are capable of storing multiple contacts they are not intended to contain information on "how" to inform these contacts, for instance, by XMPP-Message or E-Mail. The preferred way for each contact might differ. In order to achieve this, we need to enhance/blow up the structure within a event, hence one of our goals is to achieve the gathering of these information within IntelMQ in near-realtime, and not doing this in post-processing steps, which would increase the complexity of the whole architecture.

How do you mark events as "don't notify"?

Currently this is done in two ways: First: The contact might be on a Whitelist,In this case the Notification-Interval is set to -1 and a marker indicating the Whitelist entry is appended. Second: The contact doesn't receive informations due to other reasons: In this case only the Notification-Interval is set to -1.

BTW: We don't mark the Event, we mark one or more of the Contacts (plural) which have been associated to an event.

I've learned today that certats notify flag simply marks the event as "do not send it".

I'd like to stick to my proposal from #758 (comment) but the field should not contain who is contacted, instead the how is important here. See the following example:

"event": {
  <...a lot of other fields...>
  "source.abuse_contact": "mail@example.com, abuse@test.com",
  "destination.abuse_contact": "example@example.com",
  "extras": {
     "cc_url": "bla.foo.bar"
   },
   "processing": {
      "source.abuse_contact": {        /* <- Processing Info is valid for this field */
         "notification_rules": {  /* <- The specific information which was added by a bot*/
            "mail@example.com": {
                "type": "email",
                "interval": 50
                "sector": "chemical"
            },
            "abuse@test.com": {
                "type": "email",
                "interval": 0
                "sector": "tech"
            }
         }
      },
      "destination.abuse_contact": {        /* <- Processing Info is valid for this field */
         "notification_rules": {  /* <- The specific information which was added by a bot*/
            "example@example.com": {
               "type": "email",
               "whitelisted": True
            }
      }
   } 
}

In this example, three e-mail addresses were determined for an event. Two of them for source one as destination. Within an additional step, rules could be determined by an expert HOW those contacts should be informed. Those rules are the notification interval, the medium or other information. Those rules are appended to the processing-Field.

IMHO a processing field provides a simple way of extending data within the dho with additional information.

[SYNchroACK]

Concerning your proposal:
I think both parts, information from lookup bots and the blown up structure for storing abuse contact data is opposing key principles of intelmq.

Yap.

From my perspective, if there is no bot on the main repo that uses notify or other parameters structure, don't think should be added. I see this case as a good example for using the flexibility of harmonization.conf. Each organization creates their own bot to query a Contacts database and add a custom field on harmonization to suits the needs. For information sharing, I guess that fields will not be shared since its kinda an internal field(s).

Please, let's keep with the discussion #TakeItToML

From my perspective, if there is no bot on the main repo that uses notify or other parameters structure, don't think should be added.

No bot uses destination.account, *.local_ip, screenshot_url and event_hash.

[SYNchroACK]

From my perspective, if there is no bot on the main repo that uses notify or other parameters structure, don't think should be added.
No bot uses destination.account, *.local_ip, screenshot_url and event_hash.

• destination.account: source.* fields are reflected to destination.* fields for consistence purposes. Since source.account is being used, there is no justification to not have destination.accout.
• *.local_ip: inherit from abusehelper harmonization and then reflected to source and destination, , was kept for compatible translation purposes. For more details, check here.
• screenshot_url: inherit from abusehelper harmonization, was kept for compatible translation purposes
• event_hash: is useful for post-intelmq analysis of events.

@wagner-certat Do you have any other fields where you need some explanation?

@sykaeh Can you send an email to the mailing-list to share your idea and ask for comments from the community? I strongly recommend to not add this field for two main things:

• to keep IntelMQ simple for being use for other purposes
• to not use IntelMQ as a notification processor because it will add some additional issues regarding notification requirements changes during the time. This notification process (including the add of emails to which events should be send) should be done by another tool outside of IntelMQ, you call it like a IntelMQ-mailer. One scenario that this approach will solve is:

Your IntelMQ is working... you have a bot which adds the notify tag accordingly to notification criteria.... the events then are stored in your database and they are ready to be sent..... but suddenly you received a request from your manager that X type of events cannot be sent... but you already have the notifiy flag defined. You will need to do a manual UPDATE query to DB..... if you have another tool like an IntelMQ-Mailer handling this, the tool will give you the option to change the notification rules that will be applied in the moment of sending, without the need to do manual queries to database. I have more use cases where using IntelMQ pipeline to define that kind of rules can be painful in medium-long term.

But again, send the idea to mailing-list and if people want it, we will do it....