Friday, 15 May 2015

Cisco UCS C-Series servers CIMC SNMP configuration and features.

SNMP was designed as a response/request protocol. A network-management system that wants to inquire about the condition of a managed device issues a GET command to retrieve an object from the agent on the managed device. If the object is one of several objects in a list or table, the network-management system can use the GETNEXT command to retrieve the next object.

The network-managment system can also use SNMP to control the managed-device by using the SET command to change the value of an object [WARNING! SET command is NOT supported on C-Series servers]. If the managed-device needs to notify the network-management station of some event, it can issue a TRAP command to pass messages to the NMS (Network Management Station).

Cisco C-Series CIMC SNMP implementation supports SNMP version 2 and 3.Using this protocol we can monitor hardware e.g. CPU temperature etc.

The SNMP is not enabled and configured on C-Series CIMC by default. We have to enabled it and configure:

* Login to CIMC using IP address or FQDN name:

Choose -> Admin TAB -> Communications Services -> SNMP TAB -> Check BOX 'SNMP Enabled' and enter appropriate values into the fields.



* Configure your NMS (Network Managemet Station) IP address which will receive TRAPS from C-Series Server CIMC:

Choose -> Admin TAB -> Communications Services -> SNMP TAB -> Trap Destinations TAB (Right Side of the pane) -> click Add button and enter appropriate values.

 
* we can login to CIMC via SSH to check or set snmp configuration.

Under the hood of CIMC we have Linux OS with running snmpd daemon. This daemon can crash.Unfortunately to get access to Linux we have to involve Cisco TAC which will enable Linux shell using so called debug-plugin. I hope that Cisco will eventually open access to that shell following good example of Arista Networks.

We still can troubleshoot SNMP using net-snmp-utils package on your Linux box. (I am using Fedora on my laptop.)

# yum -y install net-snmp\*

The net-snmp-utils package includes a number of useful applications for querying managed devices from the command line. The snmpwalk command uses SNMP GetNext commands to recursively browse the MIB-tree below a given Table. For example, the following command will return a list of all objects subordinate to the 'cucsProcessorUnitTable'

# snmpwalk -v2c -t 15 -c public -mALL -M /home/user/Cisco_MIBs <CIMC IP ADDRESS> <OID or TEXT NAME OF OID>

-v2c : SNMP ver.2
-t : timeout 15sec
-c : community string 
-mALL : all MIBs
-M : all MIBs in directory 
 

To download Cisco UCS C-Series Manager MIBs please go to link:

ftp://ftp.cisco.com/pub/mibs/supportlists/ucs/ucs-C-supportlist.html 
 
How to translate OID to text name? We can use snmptranslate command:

If you can't remember the branch of the MIB tree that contains particular object, use command below:

#snmptranslate -IR -mALL -M /home/user/CISCO_C-Series_MIBs/UCS_Cseries2.0/ cucsProcessorUnitTable
CISCO-UNIFIED-COMPUTING-PROCESSOR-MIB::cucsProcessorUnitTable


To obtain OID of text name use command below:

#snmptranslate -On -mALL -M /home/user/CISCO_C-Series_MIBs/UCS_Cseries2.0/ CISCO-UNIFIED-COMPUTING-PROCESSOR-MIB::cucsProcessorUnitTable
.1.3.6.1.4.1.9.9.719.1.41.9 


To get text name knowing OID use this command:

#snmptranslate -mALL -M /home/user/CISCO_C-Series_MIBs/UCS_Cseries2.0/  .1.3.6.1.4.1.9.9.719.1.9.6
CISCO-UNIFIED-COMPUTING-COMPUTE-MIB::cucsComputeBoardTable

 
To print the branch of tree for particular object ID:

#snmptranslate -Tp -mALL -M /home/kb/Documents/CISCO_C-Series_MIBs/UCS_Cseries2.0/  .1.3.6.1.4.1.9.9.719.1.9.6
+--cucsComputeBoardTable(6)
   |
   +--cucsComputeBoardEntry(1)

...snip...snip...

Using the command above we can find the values of given object:

 

If you prefer GUI you can use MIB browsers e.g.

http://ireasoning.com/mibbrowser.shtml

or

https://www.manageengine.com/products/mibbrowser-free-tool/


To test your configuration you can send test trap using 'Send SNMP Test Trap' button in CIMC web console. Or if you prefer hard and unsupported way and if Cisco TAC open Linux shell entering debug-plugin you can use ipmitool and change thresholds (again this is unsupported!!! but much more fun ;-)) e.g. FAN and monitor traps on your NMS.







  the end.
 

Friday, 13 March 2015

Incremental restore of vmdk from Symantec Netbackup is corrupted.

During restore tests one of the critical MSSQL VM failed to restore from incremental Symantec Netbackup backup.  We were able to restore successfuly VM from full backup.

Restoring from incremental backup lead to corrupted state of Windows.

We checked any leftover snapshots, cleaned up the VM directory.

The key thing was VMtools version. The current ESXi build is
ESXi 5.1.0 Update 1 build-1312873. 

VMtools version installed inside affected MSSQL VM - 9.0.0.15210 – esx5.1 GA (799733), the recommended version of VMtools 9.0.5.21789 - esx5.1p03 (1312873).

Please check: http://packages.vmware.com/tools/versions

The virtual hardware version of MSSQL VM:

virtualHW.version = "8"

Based on http://blogs.vmware.com/vsphere/2013/02/clarification-on-zero-down-time-vmware-tools-uprade-in-vsphere-5-1.html

Zero-downtime VMtools upgrade is possible on virtualHW 9 or higher.

In this case reboot was required to get all new features of VMtools.


We scheduled maintenance window with MSSQL DBA, upgraded VMtools and test restore from incremental backup - it finished SUCCESSFULLY we can login into VM GuestOS without any issue.







 

Tuesday, 13 January 2015

In vSphere ESXi vmkernel.log we see messages every 30 minutes: NMP: nmp_ThrottleLogForDevice:2331 Cmd 0x85



In vmkernel.log file the ScisDeviceIO errors we are getting reported every 30 minutes:

CPUxx:32857)NMP: nmp_ThrottleLogForDevice:2331 Cmd 0x85 (0x412fc2d41c40, 34430) to dev "naa.xxxxxxxxxxxxxxxxx4a4" on path "vmhba0:C2:T1:L0" Failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:None
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x85, CmdSN 0x2ba4 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx4a4" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x4d, CmdSN 0x2ba5 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx4a4" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x1a, CmdSN 0x2ba6 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx4a4" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0

CPUxx:32857)NMP: nmp_ThrottleLogForDevice:2331 Cmd 0x85 (0x412fc2d41c40, 34430) to dev "naa.xxxxxxxxxxxxxxxxx21a" on path "vmhba0:C2:T0:L0" Failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:None
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x85, CmdSN 0x2ba7 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx21a" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x4d, CmdSN 0x2ba8 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx21a" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0
CPUxx : 32800)ScsiDeviceIO: 2337: Cmd(0x412fc2d41c40) 0x1a, CmdSN 0x2ba9 from world 34430 to dev "naa.xxxxxxxxxxxxxxxxx21a" failed H:0x0 D:0x2 P:0x0 Valid sese data: 0x5 0x20 0x0


These messages above can be safely ignored.  According to the official T10 documentation 0x85 operation code is for ATA pass-through capability:


The  underlying SCSI drives do not support SCSI2 command 0x85 in response we received SCSI Sense Code: 0x20 0x0  what means Invalid Command (for example for Seagate SCSI drives http://seagate.com/support/disc/manuals/scsi/38479j.pdf .) 

The SCSI commands 0x4d (Log Sense.) and 0x1a (Mode Sense(6)) return Invalid Command either.

The similar behaviour is described in http://kb.vmware.com/kb/1036874

IMHO that is not the best VMware KB but at least enlighten what cause the issue. It focuses on  local HP SmartArray controller  but is still valid for LSI controllers and SCSI sens code comes from disk.

===
In ESXi 5.1/5.5, you may see similar errors in the syslog.log file every 30 minutes. For example:
cpu60:16444)<4>hpsa 0000:03:00.0: Device:C4:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
cpu42:1071571)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4125c3535000, 17495) to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" on path "vmhba4:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
cpu42:1071571)ScsiDeviceIO: 2329: Cmd(0x4125c3535000) 0x85, CmdSN 0xebb from world 17495 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

These messages can be safely ignored. They appear as a result of an ATA Pass-through command which cannot be interpreted by the controller.
===

KUDOS to Jeremy Chadwick from FreeBSD mailing list

Thursday, 1 January 2015

Why hot removing running Cisco UCS blade from chassis to test vSphere HA is a bad idea.

I had a kind of unpleasant discussion with 'Senior Admin' about simulating blade failure by hot removing it from chassis. I consider it a bad idea. First of all,  hot removing blades from chassis is very poor simulation of blade failure. The blade hardware component which will have the highest probability to fail will be memory DIMM module. This is because we have more and more DIMM modules in a blade with bigger and bigger capacity per module and our world has an entropy built-in by design... I hardly believe in escaping from slots blades as a hardware failure...

The main complain of 'Senior Admin' was that after running the test 'simulating' blade failure by plugging out/plugging in a blade into chassis slot one of 8 blade went into inoperable state and required one more full power-cycle - basically required re-acknowledgement from UCSM. 'Senior Admin' believed that blades were hot-swappable and that should have never happened, particularly because this blade was going into production. Yes, blades of all mainstream vendors are hot-swappable or hot-pluggable but only from chassis perspective, it means that to remove or to add new blade you don't have to power down entire chassis. It is just common sense to power down the blade during removing to avoid any data corruption on your OS which is using this blade server and to avoid human error !!! I would think twice if some 'Senior Admin' asked me to remove running blade in datacenter with thousands blades. Dear 'Senior', believe me, they all look the same with blinking LEDs ;-) So even when you 'simulate' the failure - just simply power off and HA will kick off.

Another aspect of this test is that we live in times when hardware is cheap and people are expensive (I know nobody believes me ;-)) it is not unusual that one sysadmin manage 10k servers. Please see link below:

http://highscalability.com/blog/2013/11/19/we-finally-cracked-the-10k-problem-this-time-for-managing-se.html 
  
Dear 'Senior Admin' walking to datacenter to remove blades for 'failure simulation' before we go to production  is aaa ... stupid idea?


All vendors to replace/remove the blade from chassis require power off of blade:

1.) Removing HP BL460cG8



2.) I love IBM documentation, how to remove IBM blade:
https://publib.boulder.ibm.com/infocenter/bladectr/documentation/index.jsp?topic=/com.ibm.bladecenter.hs22.doc/dw1iu_t_removing_the_blade_server.html 


3.) How to remove Cisco B200M3 from chassis:



Hmm... but we know that UCS is slightly different and all 'brain power' of UCS is in Fabric Interconnect it is why we need to do some tasks on UCSM. 

RTFM : http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-1/b_UCSM_GUI_Configuration_Guide_2_1.html

It seems that 'Senior Admin' missed one step ;-)





BTW Cisco documentation is a little bit tricky. You can safely ignore the section 'Before You Begin'...



But what if you remove server without decommissioning it first, well you have to follow this procedure:




I would say they were lucky that 7 blades didn't require re-acknowledgment.

Okay, stop ranting - nobody is perfect even 'Senior Admin' ;-) What I suggest before putting blade into production:

1.) RTFM !!! if you really love removing blades from chassis read documentation first how to do it correctly.

2.) Before going into production burn-in blade servers all vendors have diagnostics tool, please run it e.g. Cisco blades:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/ucs_diagnostics/b_UCS_Blade_Server_Diagnostics_User_Guide/b_UCS_Blade_Server_Diagnostics_User_Guide_chapter_01.html

3.) Before putting server into production run memory test for at least 72 hours using e.g. memtest86+
http://www.memtest.org/ 

4.) We have much more elegant procedure to simulate blade failure.We can do it by triggering PSOD on ESXi host in controlled way. In ~95% cases of all  blade hardware failures ESXi host will manifest with Purple Screen Of Death PSOD e.g. DIMM failure, motherboard failure, CPU failue etc. You do not need any mechanical procedure to plugging-out/plugging-in running blade and wasting your time in unhealthy datacenter environment.

a.)    Connect to esxi host via ssh

b.)    Run from command-line:


    ~ # vsish -e set /reliability/crashMe/Panic 1

 
c.)    On KVM console you will see PSOD like below:



d.) Wait on HA event and reboot the blade or you can use the coolness of Service Profiles if you booting from SAN:

  • You can put affected ESXi host into Maintenance Mode (if you can) and then power down.
  • Disassociate Service Profile from affected blade (UCSM will attempt to gracefully shutdown or power off the blade)
  • Associate the Service Profile of affected blade with spare blade it is around 5 minutes to spin up the new blade server in UCSM.
  • Boot the server, exit from Maintenance Mode or reconnect the host in vCenter and the host pop up automatically in the vSphere cluster
  • You have the same ESXi host on different hardware that's the beauty of stateless of UCS blades.  


the end.