SNMPoTLS: Some of get or getbulk message not processed

SNMP4J
1

Hi Frank ,

We observe a issue when we send continuous get /getbulk request from client to agent few of the requests even though it is sent from agent and received in the client system it doesn’t seems to be processed from snmp4j . Due to which it waits for timeout which is set(we have it as 1 min) and then does a retry as set in the retry and in retry it gets immediately processed .
Due to this issue we are facing lot of delay in managing/supervising the agent .
The delay is reduced if we reduce the timout which is set that would cause issue when there is actual delay in network .

In the below snapshot you can observe the same data(considering the request response size ) which is received is again requested after 1 min and is processed in the next response .

We have verified agent logs and related logs in client side and see this keeps occurring frequently

image
image1861×438 37.6 KB

snmp4j logs during this time frame:

Adding operation 4 for: SocketEntry[peerAddress=135.250.142.51/10161,socket=Socket[addr=135.250.142.51/135.250.142.51,port=10161,localport=57606],lastUse=Fri Jan 09 16:49:40 IST 1970,inNetBuffer=java.nio.HeapByteBuffer[pos=811 lim=32768 cap=32768],inAppBuffer=java.nio.HeapByteBuffer[pos=0 lim=32768 cap=32768],outNetBuffer=java.nio.HeapByteBuffer[pos=0 lim=32768 cap=32768],socketTimeout=null]

Sending message with length 97 to 135.250.142.51/10161: 30:5f:02:01:03:30:10:02:03:00:d9:bf:02:03:00:80:00:04:01:04:02:01:04:04:00:30:46:04:0c:00:00:1d:3b:00:00:00:a1:87:fa:8e:33:04:00:a3:34:02:04:2e:d4:52:58:02:01:00:02:01:00:30:26:30:24:06:1f:2b:06:01:04:01:ba:3b:02:01:01:02:2d:01:01:04:0f:31:33:35:2e:32:34:39:2e:31:36:37:2e:32:32:38:02:01:02

Writing TLS outNetBuffer(PAYLOAD): java.nio.HeapByteBuffer[pos=0 lim=118 cap=32768]

Wrote TLS 118 bytes from outNetBuffer(PAYLOAD)

Payload sent completely

…
Adding operation 4 for: SocketEntry[peerAddress=135.250.142.51/10161,socket=Socket[addr=135.250.142.51/135.250.142.51,port=10161,localport=57606],lastUse=Fri Jan 09 16:50:40 IST 1970,inNetBuffer=java.nio.HeapByteBuffer[pos=539 lim=32768 cap=32768],inAppBuffer=java.nio.HeapByteBuffer[pos=0 lim=32768 cap=32768],outNetBuffer=java.nio.HeapByteBuffer[pos=0 lim=32768 cap=32768],socketTimeout=null]

Key is writable

Sending message with length 97 to 135.250.142.51/10161: 30:5f:02:01:03:30:10:02:03:00:d9:c1:02:03:00:80:00:04:01:04:02:01:04:04:00:30:46:04:0c:00:00:1d:3b:00:00:00:a1:87:fa:8e:33:04:00:a3:34:02:04:2e:d4:52:58:02:01:00:02:01:00:30:26:30:24:06:1f:2b:06:01:04:01:ba:3b:02:01:01:02:2d:01:01:04:0f:31:33:35:2e:32:34:39:2e:31:36:37:2e:32:32:38:02:01:02

Writing TLS outNetBuffer(PAYLOAD): java.nio.HeapByteBuffer[pos=0 lim=118 cap=32768]

Wrote TLS 118 bytes from outNetBuffer(PAYLOAD)

Payload sent completely

Can you please help us with issue as this is having lot of impact w.r.t time taken in managing the agent.

Please let us know if you need further details.

2

As always, I have to ask first, which SNMP4J version and which JRE version are you using?

3

Hi Frank ,

Sorry for missing to mention those details .

It is snmp4j 2.8.12, the behavior was same even in 2.8.10 .
JDK version -1.8.0_282
Regards
Vikas

4

Hi Frank,
We keep observing the below logging . Not sure if it is related to the issue which is mentioned .

INFO: Message from / not dispatched, reason: statusInfo=noError, status=-1409

5

That error code is SNMP_MP_UNKNOWN_MSGID which means that the response used a MPv3 message ID that does not match to any outstanding request.

This can be caused by:

  1. Implementation error in the agent sending the message.
  2. A late response being received after it has been timed out.
  3. Attacking or misconfigured SNMP entity sending messages with arbitrary message ID.

Maybe your agent is much slower than you think? BTW, latest Java 8 is 8u321!

6

Hi Frank,

We have verified the agent logs and tcpdump that the agent is sending back the response and is even received in client system . Have attached the snapshot of the same in my initial comment where the data is received for the request and again for same mib the request is sent after one min and then it is getting processed .

We had earlier tried check the same with snmp4j3.6 on jdk11 even there the issue was seen.

Regards,
Vikas

7

Then it is probably:

Or did you verify that the correct message ID was returned by the agent within <timeout too?

8

Hi Frank,

Unfortunately we are not able to decrypt the message to get the details of TLS packets sent. We had tried earlier and had requested for your inputs , but it had not worked even then .

Regards,
Vikas

9

Hi Vikas,
If the message is encoded and encrypted properly, then decryption with the public key of the DH key exchange provided by the sender should be easy. SNMP4J will decrypt the message too and output the BER encoded SNMP PDU to the debug log, even if the message ID is wrong.

But if the TLS session ID does not match, the message cannot be decoded, because the keys do not match. In this case connection will be closed and reopened which of course can make it worse.
Best regards
Frank