Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Lunr search indexing runs out of memory while building the docs

Description

While building our nightly documentation set we see build failures running out of memory with the JavaScript application. The Lunr indexer exceeds the X2-large memory of 16 GB RAM with our large documentation set mainly for Horizon and Meridian.

There are several ways to mitigate this issue:

  • Migrating away from a local index file strategy from Lunr to a more scalable external doc search solution, e.g. Algolia

  • Keep Lunr local and reduce the number of documentation sets especially for Horizon

In short term, we could reduce the documentation set for Horizon if would just publish the latest released version we actually support in our community. EOL doc versions, e.g. Horizon < 29 and Meridian < 2019 would be available in the archived/eol docs on vault.opennms.com.

For Meridian, we would publish the latest version for each major version which is TOG supported for the last 3 years.

We have mainly bugfix or do optimization within a major version. The documentation set is still valid and there are no differences in documentation set between Meridian 2021.1.7 and 2021.1.8. We would have also another way of showing people a) what are the latest versions we support either Horizon or Meridian and b) what versions of Meridian are active in support and which ones are EOLed.

Acceptance / Success Criteria

None

Attachments

1

Lucidchart Diagrams

Activity

Show:

Ronny Trommer January 31, 2022 at 7:46 AM

PR's merged and docs are published for 29 and Meridian 2021 accordingly.

Ronny Trommer January 18, 2022 at 11:27 AM
Edited

We decided to reduce the documentation set just for the major version numbers. We have two PR's addressing the issue and should be merged in the following order:

1. Version number change in the openms repository: https://github.com/OpenNMS/opennms/pull/4113
2. Version number changed in the opennms-prime repository: https://github.com/OpenNMS/opennms-prime/pull/24
3. Setting the branch filter to master in the docs.opennms.com repository: https://github.com/OpenNMS/docs.opennms.com/pull/14

: should be solved first.

Ronny Trommer December 17, 2021 at 11:47 PM
Edited

Just to get an idea and to make some informed decisions, I've used the default docssearch scrapper from Algolia and ran it against our current documentation set. A free tier account includes 10000 indexes for free. I ran it against each individual release and provided the numbers to get an idea about the price. The idea is you have 10000 indexes free and additional 1000 indexes cost 1 USD per month.

So as it is right now we would have ~170 USD per month for the Open Source search indexes if we would pay Algolia for the service. Any additional release of Meridian or Horizon would add ~12 USD per month.

If we qualify for the Open Source program they would run it for free. See here:
https://www.algolia.com/for-open-source/

We would probably fall into the category "Applications for non-commercial projects backed by commercial entities" which will be reviewed on a case-by-base basis.

Fixed

Details

Assignee

Reporter

Doc Backlog Status

Doc Backlog Grooming Date

Components

Affects versions

Priority

PagerDuty

Created December 17, 2021 at 11:07 PM
Updated January 31, 2022 at 7:46 AM
Resolved January 31, 2022 at 7:46 AM