Nephron: examine what additional compute and storage load is implied by the new QoS-based aggregations

Description

Goal is to investigate the impact on CPU and memory usage of the additional aggregations used for QoS queries.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Stefan Wachter April 15, 2021 at 8:55 AM
Edited

The performance requirements where met in production. therefore closing.

PS: By optimizing the aggregation calculation (cf. NMS-13100) the computational load is reduced.

Stefan Wachter January 11, 2021 at 2:23 PM
Edited

In addition to the existing 4 aggregations

  • 1: exporter/interface

  • 3: exporter/interface/(app|conv|host)

4 more aggregations are now stored:

  • 1: exporter/interface/tos

  • 3: exporter/interface/tos/(app|conv|host)

The existing aggregations are kept for backwards compatibility.

The size of the new aggregations differ from the sizes of the old aggregations by a certain factor. The upper bound for that factor is 192 (=64 * 3) because in theory there are 6 bits that distinguish DSCPs and 3 different kinds of ECNs (encoded by 2 bits). In practice however, the factor should be considerably smaller because only 18 different DSCP codes are expected. The storage cost depends on the actual existence of different DSCP codes.

If ECN information should not be used to filter/aggregate then the current implementation can be configured to ignore the ECN information. This would reduce the upper bound of the increased storage cost factor from 192 to 64.

The increase in storage cost also corresponds to increased writes to elastic.

The computational cost (in Nephron) increases, too. For the current implementation I expect the computational cost to double. This guess is based on the fact that the number of aggregations doubles. Future work may reduce the computational cost because aggregations can be build on top of each another.

 

 

Fixed

Details

Assignee

Reporter

HB Backlog Status

Components

Sprint

Fix versions

Affects versions

Priority

PagerDuty

Created October 6, 2020 at 2:49 PM
Updated May 11, 2021 at 2:38 PM
Resolved April 15, 2021 at 8:55 AM