Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Nephron: More than one FlowSummary per window/key

Description

Sometimes, Nephron creates more than one FlowSummary document per window/key. This happens repeatedly if early results are enabled.

This requires that downstream systems (e.g. ElasticSearch) have to aggregate these summaries.

Investigate why this happens and if something can be done about it.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Stefan Wachter July 28, 2021 at 3:05 PM

There are no code changes related to this ticket. Therefore I think that "fixed version" = none is correct.

Benjamin Reed July 26, 2021 at 2:33 PM

(I guess either way, the Next should go away.)

Benjamin Reed July 26, 2021 at 2:33 PM

is there an OpenNMS version this work will correlate to, since this issue is closed? Or should we remove the "Next" ?

Stefan Wachter July 23, 2021 at 11:25 AM

Under certain conditions Beam produces many early or late panes for single windows. The condition is not deterministic, for example the test RandomFlowIT.testRatesWithClockSkew always produces a massive amount of panes for exporter node 1 but only sometimes for exporter node 3. The effect seems to be caused by lateness. (The exporter nodes 1, 2, and 3 have a clock skew of -40, +40, and 0 seconds, respectively.)

It seems that sometimes panes overlap in processing time. Even after searching for detailed information on early / late panes it is not clear if this is a bug or expected behavior.

Panes can be accumulated in a post processing step. The PaneAccumulator class developed together with the Cortex sink may be used for that.

Stefan Wachter July 6, 2021 at 2:16 PM

The PaneAccumulator developed in   may be used to "post accumulate" flow summaries before they are written to ES.

Fixed

Details

Assignee

Reporter

HB Backlog Status

Doc Backlog Status

Doc Backlog Grooming Date

Sprint

Priority

PagerDuty

Created June 22, 2021 at 3:33 PM
Updated July 28, 2021 at 3:05 PM
Resolved July 23, 2021 at 11:25 AM