Cleanup Interfaces Tagged for Flows

Description

Currently as flows are processed, the interfaces are tagged in the DB as having flow.

If a particular interface is no longer sending flows, this interface is still tagged. No method of cleanup exists for interfaces.

Possible solutions:
1) Run expensive ES query on weekly/monthly basis for all interfaces with flows, compare and cleanup against database.
2) Add timestamp adjacent to flow tag in database, update field periodically to prevent flooding of DB timestamp updates. Create parameter that removes tag from interfaces if date is greater than X.
3) Similar to solution 2, run ES query for netflow indexes and calculate X from oldest index date.
4) Convert `hasflows` field in DB to date field updating periodically to represent last time flows seen.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Christian Pape February 12, 2020 at 6:42 AM

Merged.

Jesse White January 22, 2020 at 2:16 PM

I'd like to avoid having to perform any active queries to do this logic, so let's try and find a way to make it work passively as flows are processed.

Instead of a single boolean hasflows field on the snmpinterfaces table I suggest we split this up into two nullable date fields: last_ingress_flow and last_egress_flow

We can update these timestamps once an hour (or similar) to minimize the DB impact.

The logic used to determine whether or not an interface has flows (and should be marked in the UI or made available in Grafana) can then be made configurable:

  • Require ingress/egress: yes/no

  • Time period i.e. 7d

Chen Khin Wai November 28, 2019 at 12:53 AM

Any updates on this bug?

Fixed

Details

Assignee

Reporter

Labels

Components

Sprint

Fix versions

Affects versions

Priority

PagerDuty

Created September 11, 2019 at 11:09 PM
Updated February 12, 2020 at 6:42 AM
Resolved February 12, 2020 at 6:42 AM