Troubleshooting SNMPv3

This article will outline the major steps required to check and troubleshoot some simple defects in SNMPv3. The reason why I am specifically focusing on SNMPv3 is the fact that since its version 3, SNMP has introduced authentication and encryption, often referred to "Auth" and "Ident". Bear in mind that this article does not contain step-by-step full procedure to configure SNMP, only outlines some basic errors and ways how to diagnose them.
To start with, one needs to create a user. Before that we will stop the snmpd service and only create the user after the service has been stopped.

This is shown in the Figure above. Note that the user requires both password for authentication ("authPassword1") , hashing algorithm for password ("SHA"), encryption pre-shared key ("encryptPassword1") and symmetric encryption algorithm ("AES").  The next step is checking if the restarted SNMP service is listening on its default (161) UDP port.
From the remote machine the snmp query has been sent via snmpwalk, specifying all the required SNMPv3 parameters explained above.

We see that snmpwalk produces the list of values, so all seems to work ok. One crucial difference of the SNMPv3 to SNMPv2c or 1 is the integrity protection, in other words, SNMP transmission that is encrypted by the symmetric AES key (encryptPassword1) in our case. To check the transmission data, wireshark is launched to monitor the traffic.

In yellow are the most important elements that indicate encryption. Note in the SNMP protocol header the "msgVersion" which states "3" (SNMPv3). If we really want to be sure the snmp response is ok, we can import the decryption key (remember, it's the symmetric encryption here, so we can use the same key to both encrypt and decrypt traffic) into the Wireshark and have a look at the actual SNMP payload.

The figure above shows the edit for the SNMP protocol->"Edit" and specifying the username, authentication model, password, encryption algorithm and encryption pre-shared key. After this operation, Wireshark will add a tab for decrypted traffic.

Note in the above figure in the ASCII representation (right bottom side) clear text output.
At this stage, another SNMP query was launched but this time with the incorrect encryption key - the idea here was to investigate if a traffic capture will show any indication of an invalid encryption key.

To my surprise, the wireshark showed a malformed packet. Not sure why this happened as UDP malformed packets are usually reported if they contain invalid UDP checksum. But, then I remember that malformed DNS packets will show similar output (malformed packet). So not sure if wireshark actually tries to interpret the SNMP context , after all, it has the protocol specific dissector. Whatever the reason for this may be, we still cannot determine from the above output what exactly is the error. Running debugging process on the snmpwalk of the SNMP query initiator we can get some more insight into the potential root cause.

The red rectangle shows at its top "sc_generate_keyed_hash()" and then resending the message. The rationale here seems to be an invalid hash (remember, hash function is used for hashing the password for authentication) because after that the retransmit occurs. However, this is not the root cause in our case as we have supplied the incorrect encryption key. However, note that the debug does not explicitly state that there is an invalid hash or any type of error - it simply outlines that the last function done by the snmpwalk was hashing before the retransmission happened.
The only conclusion here is that this output only shows the last successfully executed function (hashing) before the retransmit occurred (due to an invalid encryption key). The whole debugging is triggered by the switch "-D ALL".  The next example shows what happened when the invalid authentication password was supplied.

This one is fairly obvious - an output clearly indicates the error spot. With access to the SNMP service, we can consult the log file to obtain the same result.

Next thing to inspect was SNMP Trap , also in its version 3. On the SNMP service daemon side , we need to configure some basic stuff for SNMP Trap.

Everything seems pretty obvious, except for somewhat obscure parameter of "EngineID". EngineID is specific to the SNMPtrap sending machine and must be configured in the SNMP TRap receiver. See http://www.net-snmp.org/tutorial/tutorial-5/commands/snmptrap-v3.html for more details. The min thing is that EngineID is automatically derived by snmpwalk, but in case of SNMP Trap one needs to manually configure it. EngineID is usally found in the snmpapp.conf file on your Linux system. Once you get the correct EngineID, put it into the snmptrap (or make sure the SNMP Trap collector has it configured, in which case you may omit the engineID parameter from snmptrap).

This will result in the output on the SNMP Trap daemon showing the details of the trap.