harmonization: move the user_agent info out from extra and make it it's own proper field

[aaronkaplan]

user_agent is quite common , please make it it's own outright field.

[sebix]

Has already been discussed in #185 and the conclusion was that it will be moved to extra.

[aaronkaplan]

i am still not sure if I agree.

[aaronkaplan]

-> @dmth ping.
I'd like to put user_agent into the DHO. It is important enough and used often enough for botnet <-> C&C traffic to actually have it.

[dmth]

First of all, I've not read #185. Should do it, though.

I'm really not sure if user_agent is worth a field in the DHO.
I don't know how many feeds provide this information.

In general:
We have to be aware, that each additional field in the DHO creates one migration which is necessary for the database. This has to be communicated with the community in a transparent way. For instance:

On first of August the DHO will have changed. The following fields were added:

• a
• b

One field was renamed:

• c

Please run scripname.py -database D -table t in order to update your postgresql database from > the latest version (since $commit) to the new one.

To keep this "notification overhead" as minimal as possible we should collect changes which concern the DHO and integrate them in one step.

[aaronkaplan]

On 25 Jul 2016, at 16:38, Dustin Demuth notifications@github.com wrote:

First of all, I've not read #185. Should do it, though.

I'm really not sure if user_agent is worth a field in the DHO.
I don't know how many feeds provide this information.

They do. In general many HTTP based bot <-> C&C sinkhole feed actually provide the user-agent.

In general:
We have to be aware, that each additional field in the DHO creates one migration which is necessary for the database. This has to be communicated with the community in a transparent way. For instance:

Sure.

Got it...
But on the other hand, we are not in release 1.0 yet. So we don't need to maintain backwards compatibility yet (in 1.0 we will).

@aaronkaplan Do you want to do this before 1.0 or can we postpone this?

[aaronkaplan]

timeout. Let's leave it as is (in the extra). Move this to 1.1