Provide the ability in Nephron to write to Cortex

Description

This feature will track the work required to modify Nephron to add Cortex as an option for database in addition to the current ES, and the ability to write to Cortex.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Stefan Wachter July 20, 2021 at 3:25 PM

Stefan Wachter July 14, 2021 at 1:49 PM

Pane accumulation alone was not enough. In case of early / late results it can happen that an early result pane of a later window completes before a late result pane of an earlier window.

A stateful transform in the global window is used to assign unique index numbers to results for the same event timestamp. That index is used as an additional metrics label in Cortex.

Stefan Wachter July 6, 2021 at 2:08 PM
Edited

Cortex requires that metric values are written ordered by timestamps (i.e. older values must not be written after newer values have been written). Beam however, does not guarantee processing order, i.e. windows (and panes) may be processed in parallel and "older" results may complete earlier than "newer" results.

I implemented a stateful PaneAccumulator transformation that can accumulate values in several panes of the same window into one or more output values. Accumulated values are tupled with a sequential index number that can be used as an additional label for cortex metrics. When metrics are retrieved cortex can sum over all metrics that differ in that index only.

It has to be tested how PaneAccumulator influences performance. There is some additional state to be stored per window/key and some additonal shuffling done.

Fixed

Details

Assignee

Reporter

HB Backlog Status

Sprint

Fix versions

Priority

PagerDuty

Created June 15, 2021 at 8:12 PM
Updated September 8, 2021 at 2:41 PM
Resolved September 8, 2021 at 2:41 PM