Currently as flows are processed, the interfaces are tagged in the DB as having flow.
If a particular interface is no longer sending flows, this interface is still tagged. No method of cleanup exists for interfaces.
Possible solutions: 1) Run expensive ES query on weekly/monthly basis for all interfaces with flows, compare and cleanup against database. 2) Add timestamp adjacent to flow tag in database, update field periodically to prevent flooding of DB timestamp updates. Create parameter that removes tag from interfaces if date is greater than X. 3) Similar to solution 2, run ES query for netflow indexes and calculate X from oldest index date. 4) Convert `hasflows` field in DB to date field updating periodically to represent last time flows seen.
Acceptance / Success Criteria
None
Lucidchart Diagrams
Activity
Show:
Christian Pape February 12, 2020 at 6:42 AM
Merged.
Jesse White January 22, 2020 at 2:16 PM
I'd like to avoid having to perform any active queries to do this logic, so let's try and find a way to make it work passively as flows are processed.
Instead of a single boolean hasflows field on the snmpinterfaces table I suggest we split this up into two nullable date fields: last_ingress_flow and last_egress_flow
We can update these timestamps once an hour (or similar) to minimize the DB impact.
The logic used to determine whether or not an interface has flows (and should be marked in the UI or made available in Grafana) can then be made configurable:
Currently as flows are processed, the interfaces are tagged in the DB as having flow.
If a particular interface is no longer sending flows, this interface is still tagged. No method of cleanup exists for interfaces.
Possible solutions:
1) Run expensive ES query on weekly/monthly basis for all interfaces with flows, compare and cleanup against database.
2) Add timestamp adjacent to flow tag in database, update field periodically to prevent flooding of DB timestamp updates. Create parameter that removes tag from interfaces if date is greater than X.
3) Similar to solution 2, run ES query for netflow indexes and calculate X from oldest index date.
4) Convert `hasflows` field in DB to date field updating periodically to represent last time flows seen.