-
Notifications
You must be signed in to change notification settings - Fork 0
Filter Layer
The filter layer is a user defined operation that is performed on a list of AggregateTraces. Each AggregateTrace represents a unique combination of protocol, port and IP address between two hosts. Initially this list is retrieved from the database and then processed through the layers defined in a given layout. The filter layer operates on this list and filters in or out items that fulfil its internal requirements.
A filter list in its simplest form is an ILayer with an extended FilterList, it is this FilterList that holds all the data relevant for filtering and can be reused by other layers for the selection of AggregateTraces.
The structure in code (shortened for brevity):
public class FilterLayer : ILayer
{
public string Type { get; set; } = LayerType.Filter;
public string Name { get; set; }
public bool Enabled { get; set; }
public FilterList FilterList { get; set; }
}When a FilterCondition is applied it tests whether it matches against a given combination of source and destination address only if it matches it will act upon the Include field. A match is concluded when the AggregateTrace.SourceAddress masked with the SourceAddressMask is the same as the SourceAddress and the same is true for the AggregateTrace.DestinationAddress.
Should a source port or destination port be set then they will be tested against too. Same goes for the protocol type.
Example:
This example showcases how a single FilterCondition can be constructed to match any AggregateTrace with the source of 172.168.0.0/16 and destination any value.
The FilterCondition that is used during the examples below:
FilterCondition {
SourceAddress: 172.168. 0. 0
SourceAddressMask: 255.255. 0. 0
SourcePort: null
DestinationAddress: 0. 0. 0. 0
DestinationAddressMask: 0. 0. 0. 0
DestinationPort: null
}Matching with an AggregateTrace that will match:
AggregateTrace {
SourceAddress: 172.168. 23. 12
DestinationAddress: 10 . 34.123. 40
}
// #1: Apply the mask to the source address
AggregateTrace.SourceAddress: 172.168. 23. 12
FilterCondition.SourceAddressMask: 255.255. 0. 0
AND Operation: ===============
Result: 172.168. 0. 0
// #2: Compare the result with the FilterCondition.SourceAddress
Result: 172.168. 0. 0
FilterCondition.SourceAddress: 172.168. 0. 0
===============
Match: YES
// #3: Repeat step #1 and #2 for the destination address
AggregateTrace.SourceAddress: 172.168. 23. 12
FilterCondition.SourceAddressMask: 0. 0. 0. 0
AND Operation: ===============
Result: 0. 0. 0. 0
FilterCondition.SourceAddress: 0. 0. 0. 0
===============
Match: YES
// Result: The given AggregateTrace matches this FilterCondition and will be included or excluded based on the FilterCondition.Include value.Matching with an AggregateTrace that will not match:
AggregateTrace {
SourceAddress: 192.168. 1. 100
DestinationAddress: 10 . 34.123. 40
}
// #1: Apply the mask to the source address
AggregateTrace.SourceAddress: 192.168. 1. 100
FilterCondition.SourceAddressMask: 255.255. 0. 0
AND Operation: ===============
Result: 192.168. 0. 0
// #2: Compare the result with the FilterCondition.SourceAddress
Result: 192.168. 0. 0
FilterCondition.SourceAddress: 172.168. 0. 0
===============
Match: NO
// As the first check has failed this AggregateTrace is no match and is ignored.Matching ports and protocols is done by setting any of SourcePort, DestinationPort and/or Protocol. If they are set the FilterCondition will match against it.
// Only match traces which are HTTP and originate from a specific server pool
FilterCondition {
SourceAddress: 172.168. 32. 0
SourceAddressMask: 255.255.255. 0
SourcePort: 80
DestinationAddress: 0. 0. 0. 0
DestinationAddressMask: 0. 0. 0. 0
DestinationPort: null
Protocol: Tcp
}
// This is a match because
// 1. Both SourceAddress and DestinationAddress match the FilterCondition
// 2. The FilterCondition.SourcePort is not null AND
// FilterCondition.SourcePort equals AggregateTrace.SourcePort
// 3. The FilterCondition.Protocol is not null AND
// FilterCondition.Protocol equals AggregateTrace.Protocol
AggregateTrace {
SourceAddress: 172.168. 32.113
SourcePort: 80
DestinationAddress: 10 . 34.123. 40
DestinationPort: 238
Protocol: Tcp
}
// This is not a match because
// 1. FilterCondition.SourcePort is not null BUT
// FilterCondition.SourcePort is not equal to AggregateTrace.SourcePort
AggregateTrace {
SourceAddress: 172.168. 32.100
SourcePort: 443
DestinationAddress: 10 . 34.123. 40
DestinationPort: 231
Protocol: Tcp
}These examples demonstrate how a single FilterCondition can match against a single AggregateTrace however when filtering a list we want to apply an entire FilterList to a list of AggregateTraces. A single FilterCondition is not capable of sorting a list or filtering items. The only public method it exposes is whether it matches against a given AggregateTrace.
The FilterList object consists of a list of conditions, each FilterCondition. A FilterList is designed to take in a list of AggregateTraces and remove or include objects based on its FilterConditions.
The structure in code (shortened for brevity):
public class FilterList
{
public bool ImplicitInclude { get; set; }
public List<FilterCondition> Conditions { get; set; }
}
public class FilterCondition
{
public byte[] SourceAddress { get; set; }
public byte[] SourceAddressMask { get; set; }
public ushort? SourcePort { get; set; }
public byte[] DestinationAddress { get; set; }
public byte[] DestinationAddressMask { get; set; }
public ushort? DestinationPort { get; set; }
public bool Include { get; set; }
}The FilterList is constructed of a list of FilterConditions. It filters a list of AggregateTraces by looping through all and applying all FilterConditions to every AggregateTrace until on matches. Upon a match the FilterList will then either include the AggregateTrace in the result list or exclude it. Should no FilterCondition match the FilterList has the ImplicitInclude property which if true includes all non-match AggregateTraces or excludes them.
This pseudo code example is the algorithm, it is shortened and simplified for brevity:
for aggregateTrace in aggregateTraces {
for filterCondition in filterConditions {
if filterCondition.Match(aggregateTrace) {
if filterCondition.Include {
resultList.Add(aggregateTrace);
} else {
// Do not add the aggregateTrace to the result list i.e. exclude it
}
goto nextAggregateTrace; // In any case skip to the next AggregateTrace
} else {
continue; // Try the next filter condition
}
}
// Should the loop end normally then no FilterCondition has caused a match
if (ImplicitInclude) {
resultList.Add(aggregateTrace);
} else {
// Do not add the aggregateTrace to the result list i.e. exclude it
}
label: nextAggregateTrace;
}NOTE: For brevity port and protocol matching are excluded from the examples below.
A FilterList can be represented as the table below, reminiscent of an access control list in networking:
| Idx | Source | Destination | Action |
|-----|--------------------|--------------------|---------|
| #1 | 172.168. 15. 30/32 | 0. 0. 0. 0/32 | EXCLUDE | // Filter Condition #1
| #2 | 172.168. 0. 0/16 | 0. 0. 0. 0/32 | INCLUDE | // Filter Condition #2
| | 0. 0. 0. 0/0 | 0. 0. 0. 0/0 | EXCLUDE | // The implicit condition, its a catch all that makes sure every case is covered
Example:
In this example a list of three AggregateTraces is filtered using the above FilterList.
aggregateTraces = [
AggregateTrace {
SourceAddress: 172.168. 15. 30
DestinationAddress: 10. 0. 0. 1
}
AggregateTrace {
SourceAddress: 172.168.100. 20
DestinationAddress: 192.168. 1. 1
}
AggregateTrace {
SourceAddress: 234. 32.193.167
DestinationAddress: 10. 20. 42. 23
}
]The first AggregateTrace is matched by the first FilterCondition. Based on the EXCLUDE action this AggregateTrace will be excluded from the result list.
172.168. 15. 30 10. 0. 0. 1
| #1 | 172.168. 15. 30/32 | 0. 0. 0. 0/32 | EXCLUDE | // MATCH <= Src: MATCH, Dst: MATCH
| #2 | 172.168. 0. 0/16 | 0. 0. 0. 0/32 | INCLUDE | // NOT COMPARED
| | 0. 0. 0. 0/0 | 0. 0. 0. 0/0 | EXCLUDE | // NOT COMPARED
Result: EXCLUDE
The second AggregateTrace matches the second FilterCondition. The INCLUDE action of this condition means this AggregateTrace will be included in the result list.
172.168.100. 20 192.168. 1. 1
| #1 | 172.168. 15. 30/32 | 0. 0. 0. 0/32 | EXCLUDE | // NO MATCH <= Src: NO MATCH, Dst: MATCH
| #2 | 172.168. 0. 0/16 | 0. 0. 0. 0/32 | INCLUDE | // MATCH <= Src: MATCH , Dst: MATCH
| | 0. 0. 0. 0/0 | 0. 0. 0. 0/0 | EXCLUDE | // NOT COMPARED
Result: INCLUDE
The third AggregateTrace does not match either the first or second FilterCondition, so it falls to the implicit condition, which is an EXCLUDE action. Therefore, this AggregateTrace will also be excluded from the result list.
234. 32.193.167 10. 20. 42. 23
| #1 | 172.168. 15. 30/32 | 0. 0. 0. 0/32 | EXCLUDE | // NO MATCH <= Src: NO MATCH, Dst: MATCH
| #2 | 172.168. 0. 0/16 | 0. 0. 0. 0/32 | INCLUDE | // NO MATCH <= Src: NO MATCH, Dst: MATCH
| | 0. 0. 0. 0/0 | 0. 0. 0. 0/0 | EXCLUDE | // MATCH <= Src: MATCH , Dst: MATCH
Result: EXCLUDE
The API allows the insertion of new layers as well as the movement, deletion, updating and reading of existing layers. Specific endpoints can be found in the Swagger UI.
Example:
This example showcases the required JSON structure to be able to create a layer via the API. This specific JSON would create a layer which would remove any HTTP traffic originating from or going to the subnet 10.37.0.0/16 which the user has called Remove Site A. To do this the user has to create two different conditions.
{
"name": "Exclude Site A", // User defined name
"type": "filter", // Required to create a filter type layer
"enabled": true, // Whether the layer should be en- or disabled
"filterList": { // Holds the data to construct the filter list
"conditions": [
{
"sourceAddress": "10.37.0.0", // Match traffic originating from Site A
"sourceAddressMask": "255.255.0.0",
"sourcePort": 80,
"destinationAddress": "0.0.0.0",
"destinationAddressMask": "0.0.0.0",
"destinationPort": null,
"protocol": "Tcp",
"include": false
},
{
"sourceAddress": "0.0.0.0",
"sourceAddressMask": "0.0.0.0",
"sourcePort": null,
"destinationAddress": "10.37.0.0", // Match traffic inbound to Site A
"destinationAddressMask": "255.255.0.0",
"destinationPort": 80,
"protocol": "Tcp",
"include": false
}
],
"implicitInclude": true // Keep all other aggregate traces
}
}