EMC XtremeIO with VMware View very high CPU usage on ESXi hosts

New vBlock for extreme applications, 4 Cisco B200 M3 blades, 1000 VMs are deployed - CPU on all blades hit the roof. We didn't expect that.

 In XtremIO logs we notice a huge amount of SCSI reservation logs:

<info>2014-06-11 15:09:59.507783 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:5f, vol=12, opcode=42
<info>2014-06-11 15:09:59.705085 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=12, opcode=42
<info>2014-06-11 15:10:00.314514 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=6, opcode=42
<info>2014-06-11 15:10:00.973382 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=42
<info>2014-06-11 15:10:00.973393 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 15:10:01.088004 localhost xtremapp: [8474(9775 nb_truck_5)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=10, opcode=42
<info>2014-06-11 15:13:53.936724 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 16:24:34.787011 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=16, opcode

Checking VAAI settings and we see human-error:

~ # esxcfg-info -o | egrep -B 8 '/HardwareAcceleratedMove|/HardwareAcceleratedLocking'
            \==+Advanced Integer Option :
               |----Option Name.....................................HardwareAcceleratedLocking
               |----Current Value...................................0
               |----Default Value...................................1
               |----Min Value.......................................0
               |----Max Value.......................................1
               |----Hidden..........................................false
               |----Parent........................................../VMFS3/
               |----Path............................................/VMFS3/HardwareAcceleratedLocking
--
            \==+Advanced Integer Option :
               |----Option Name.....................................HardwareAcceleratedMove
               |----Current Value...................................0
               |----Default Value...................................1
               |----Min Value.......................................0
               |----Max Value.......................................1
               |----Hidden..........................................false
               |----Parent........................................../DataMover/
               |----Path............................................/DataMover/HardwareAcceleratedMove

 Enabling VAAI primitives:

 

# esxcli system settings advanced set -o /DataMover/HardwareAcceleratedMove -i 1

# esxcli system settings advanced set -o /VMFS3/HardwareAcceleratedLocking -i 1

 

SCSI reservations disappear from XtremeIO logs. 

 

pCPUs utilisation around 50% during deployment 1000 VMs. After ~2hrs 1000 desktops ready to use. That's really extreme!!!

 

Remember to get extreme performance keep VAAI primitives enabled on vSphere hosts !!!  

the end.