All work

Select view

List view

Detail view

Select search mode

Basic

JQL

Bouncycastle JARs break large-key crypto operations
NMS-7959
Resolved issue: NMS-7959
Hung threads causing 100% CPU usage (possibly Jetty + HTTPS/SSL related)
NMS-4940
Resolved issue: NMS-4940

2 of 2

Bouncycastle JARs break large-key crypto operations

Fixed

Description

When trying to use the org.opennms.core.web.HttpClientWrapper class to make an HTTPS client connection to a server supporting high-strength cipher suites, I get the following exception stack trace:

javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[?:1.8.0_65]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:290) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:259) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:125) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:319) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.3.6.jar:4.3.6]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) ~[httpclient-4.3.6.jar:4.3.6]
	at org.opennms.netmgt.notifd.MattermostNotificationStrategy.send(MattermostNotificationStrategy.java:111) [opennms-services-18.0.0-SNAPSHOT.jar:?]
	at org.opennms.netmgt.notifd.ClassExecutor.execute(ClassExecutor.java:69) [opennms-services-18.0.0-SNAPSHOT.jar:?]
	at org.opennms.netmgt.notifd.NotificationTask.run(NotificationTask.java:269) [opennms-services-18.0.0-SNAPSHOT.jar:?]
Caused by: java.lang.RuntimeException: Could not generate DH keypair
	at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:81) ~[?:1.8.0_65]
	at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:1.8.0_65]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:1.8.0_65]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:1.8.0_65]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:1.8.0_65]
	... 16 more
Caused by: java.security.InvalidAlgorithmParameterException: parameter object not a ECParameterSpec
	at org.bouncycastle.jce.provider.JDKKeyPairGenerator$EC.initialize(Unknown Source) ~[bcprov-jdk14-1.38.jar:1.38.0]
	at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:76) ~[?:1.8.0_65]
	at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:1.8.0_65]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:1.8.0_65]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:1.8.0_65]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:1.8.0_65]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:1.8.0_65]
	... 16 more

The bouncycastle JCE provider appears to be taking precedence over the JDK-provided one, resulting in this problem. Removing the bcprov JAR from OPENNMS_HOME/lib enables the connections to succeed.

We appear to have picked up an undeclared dependency on the following artifacts from group bouncycastle

bcmail-jdk14
bcprov-jdk14
bctsp-jdk14

We have existing exclusions in the dependencies/jasper POM for bcmail and bcprov. Somebody thought iText could be the culprit.

Environment

Fedora 21 (4.1.8-100.fc21.x86_64 #1 SMP Tue Sep 22 12:13:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux) Oracle JDK 1.8_65

Acceptance / Success Criteria

None

Linked work items

blocks

NMS-9416

EAP-TTLS tunneling protocol for RADIUS poller/detector

Lucidchart Diagrams

Details
Assignee
Benjamin Reed
Reporter
Jeff Gehlbach
Labels
httpsssl
Components
Build / Packaging
Sprint
None
Fix versions
16.0.4
Meridian-2015.1.1
17.0.0
18.0.0
Affects versions
16.0.3
Meridian-2015.1.1
17.0.0
Priority
Blocker

PagerDuty

Created November 3, 2015 at 10:38 AM

Updated October 16, 2017 at 1:16 PM

Resolved November 6, 2015 at 11:13 AM

Activity

Ronny TrommerApril 1, 2016 at 8:15 PM

@Benjamin Reed seems to be fixed, can we delete the associated branch? https://github.com/OpenNMS/opennms/tree/jira/NMS-7959

Benjamin ReedNovember 6, 2015 at 11:13 AM

This got fixed in foundation and has merged forward to all relevant branches.

Ronny TrommerNovember 4, 2015 at 3:44 AM

It is possible this issue has also effect when you try to SSH in Karaf. The SSH connection breaks with connection refused. The ssh -v output looks like the following:

OpenSSH_6.9p1, LibreSSL 2.1.7
..
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version SSHD-CORE-0.12.0
debug1: no match: SSHD-CORE-0.12.0
debug1: Authenticating to localhost:8101 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 127.0.0.1

In Karaf you can see following Exception during the login:

2015-11-04 09:37:55,231 INFO  org.apache.sshd.core:0.12.0(24) [sshd-SshServer[709fa416]-nio2-thread-6] org.apache.sshd.server.session.ServerSession: Server session created from /127.0.0.1:49595
2015-11-04 09:37:55,231 INFO  org.apache.sshd.core:0.12.0(24) [sshd-SshServer[709fa416]-nio2-thread-6] org.apache.sshd.server.session.ServerSession: Kex: server->client aes128-ctr hmac-sha1 none
2015-11-04 09:37:55,231 INFO  org.apache.sshd.core:0.12.0(24) [sshd-SshServer[709fa416]-nio2-thread-6] org.apache.sshd.server.session.ServerSession: Kex: client->server aes128-ctr hmac-sha1 none
2015-11-04 09:37:55,231 WARN  org.apache.sshd.core:0.12.0(24) [sshd-SshServer[709fa416]-nio2-thread-6] org.apache.sshd.server.session.ServerSession: Exception caught
java.security.InvalidAlgorithmParameterException: parameter object not a ECParameterSpec
	at org.bouncycastle.jce.provider.JDKKeyPairGenerator$EC.initialize(Unknown Source)[bcprov-jdk14-1.38.jar:1.38.0]
	at java.security.KeyPairGenerator.initialize(KeyPairGenerator.java:411)[:1.8.0_45]
	at org.apache.sshd.common.kex.ECDH.getE(ECDH.java:58)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.server.kex.AbstractDHGServer.init(AbstractDHGServer.java:71)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:359)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:731)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:187)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:173)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
	at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_45]
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[24:org.apache.sshd.core:0.12.0]
	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_45]
	at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.8.0_45]
	at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.8.0_45]
	at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)[:1.8.0_45]
	at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)[:1.8.0_45]
	at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)[:1.8.0_45]
	at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:173)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:129)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:108)[24:org.apache.sshd.core:0.12.0]
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
	at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_45]
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[24:org.apache.sshd.core:0.12.0]
	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_45]
	at sun.nio.ch.Invoker$2.run(Invoker.java:218)[:1.8.0_45]
	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)[:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745)[:1.8.0_45]

Benjamin ReedNovember 3, 2015 at 1:32 PM

I believe I have the fix for this, just testing it now.

All work

Bouncycastle JARs break large-key crypto operations

Description

Environment

Acceptance / Success Criteria

Linked work items

blocks

Lucidchart Diagrams

DetailsAssigneeBenjamin ReedBenjamin ReedReporterJeff GehlbachJeff GehlbachLabelshttpssslComponentsBuild / PackagingSprintNone+1Fix versions16.0.4Meridian-2015.1.117.0.018.0.0Affects versions16.0.3Meridian-2015.1.117.0.0PriorityBlocker

Details

Assignee

Reporter

Labels

Components

Sprint

Fix versions

Affects versions

Priority

PagerDutyPagerDuty Incident

PagerDuty

Activity

Ronny TrommerApril 1, 2016 at 8:15 PM

Benjamin ReedNovember 6, 2015 at 11:13 AM

Ronny TrommerNovember 4, 2015 at 3:44 AM

Benjamin ReedNovember 3, 2015 at 1:32 PM

Details
Assignee
Benjamin Reed
Reporter
Jeff Gehlbach
Labels
httpsssl
Components
Build / Packaging
Sprint
None
Fix versions
16.0.4
Meridian-2015.1.1
17.0.0
18.0.0
Affects versions
16.0.3
Meridian-2015.1.1
17.0.0
Priority
Blocker

PagerDuty