ATS - Atomic Test&Set primitive.
SCSI2 reservation - entire volume is locked
ATS - lock only block
VMFS3: layout - resources (inodes, blocks, sub-blocks etc.)organized into clusters, each cluster has an associated lock and metadata. Clusters form a cluster groups. Cluster groups repeat to make the filesystem.
ATS Values:
* space locality - esxi hosts strive for contiguous blocks of objects they manage
* reduces storage resources contention
* larger VMFS datastores size
* higher VMs density
* reduced datastore management
ATS Caveats:
* vmotion vs. storage vMotion e.g.
1.) on esxi1 we create 10 VMs which see 500GB at the beginning of 1TB volume
2.) host esxi1 is accessing this contiguous area
3.) on esxi2 we create VM which allocate 100GB on the same 1TB volume at the end of 500GB area accessed by esxi1. Both hosts work fine without any storage contention.
4.) now, we vmotion 5 VMs to esxi2,
5.) esxi2 host get access to contiguous space previously accessed by esxi1
6.) now, this hosts start contending for the resources in different area of the disk. Space locality is disturbed.
Do vMotion only when is needed e.g. DRS recommendation. Instead of causing multiple hosts access region on the disk which other hosts will need and increasing the potential for contention of these resources. Storage vMotion more preferable than classic vMotion.
It doesn't mean don't use vmotion !!! It means think when you scale out with vSphere cluster !!!
* free capacity (filling up VMFS)- need free capacity on the datastore, esxi don't like full datastores ;)
* storage vMotion -> force esxi to find contiguous space.
UNMAP primitive:
# esxcli system settings advanced set -i 0 -o /VMFS3/EnableBlockDelete
# vmkfstools -y X (X = 1 - 100% default 60%)
VMware disabled UNMAP - only manual reclaiming posssible.
Reclaiming space => huge impact on performance. UNMPAP operation create a balloon file equal to size of space to be de-allocated. Balloon file causes many WR I/O.
Avoid X > 90% !!!
You may not get back space which you expected to get back !
Percentage is derived from free space available on VMFS datastore e.g.
2TB datastore 50% full with vmkfstools -y 50 will attempt to reclaim 0.5TB (512GB) = 50% of 1TB free.
VAAI Consideration/Hidden Benefits:
* fewer commands send to the storage array.
UNMAP command is different and depends on array implementation, it could be more command than without UNMAP primitive.
Some VAAI implementation will NOT work if datastore is not aligned:
* VMFS3 datastore to 64kB
* VMFS5 datastore to 1MB
* some arrays may reject non-aligned VAAI operations
* concurrent clone operations:
-- maybe throttle by array at some threshold (current backend processing limit)
-- limit number of concurrent clone/zero to 3 or 4.
VAAI and array replication - some performance deprecation for VAAI.
Fixed I/O path policy is NOT ALUA aware and not recommended for ALUA arrays.
Changing MPIO settings for multiple LUNs:
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp device set -P VMW_PSP_RR -d $i; done
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp psp roundrobin deviceconfig set -t iops –I 1 -d $i; done
FIXED_AP - ALUA aware
ESXi5 rolled FIXED_AP functionality into the FIXED I/O path policy. FIXED_AP only applicable to ALUA capable arrays.
Is FIXED in ESXi5 recommended for ALUA arrays?
NO ! For general usage (LUN thrashing). YES ! as a tool for quickly restoring balance in an unbalanced array LUN configuration.
Round Robin - lower IOPS is better, high IOPS value for sequential workload.
Friday, 15 August 2014
Tuesday, 12 August 2014
No access to VM console; Unable to connect to the MKS: A general system error occured: Internal error
All 16 hosts in cluster are up and running since long time without any issue - uptime 300+ days. On all hosts we cannot get access to VM console. Opening VM console from viClient we get error "Unable to connect to the MKS: A general system error occured: Internal error".
We cannot vmotion VMs to another esxi hosts in cluster.
We login into esxi hosts and noticed that root Ramdisk is full:
# vdf -h | tail -6
The uptime of esxi hosts was impressive:
# uptime
When we tried to get information about Virtual Machines using vim-cmd command we got error:
# vim-cmd vmsvc/getallvms
We tried to figure out what consumed space on root in Ramdisk, we run command:
# find / -size +10k -exec du -h {} \; | egrep -v volumes | egrep -v disks | less
I spotted a lot of EMCProvider logs in /opt/emc/cim/log
# ls -l | head -5
And bingo! these logs eat the space:
# du -h /opt/emc/cim/log/
It seems that EMCProvider logs haven't rotated and fulfilled root in Ramdisk. I couldn't find any parameter in conf file to setup rotation of EMCProvider logs - it is more feature than bug ;)
We deleted logs older than 200 days (eventually we deleted all EMCProvider logs older than 1 day) on esxi hosts in cluster:
# cd /opt/emc/cim/log/
# find . -name '*.log' -mtime +200 -exec rm -f {} \;
We got some free space on root and were able to got access to some VM console, but some VMs started to show another error 'Unable to connect to the MKS: Failed to connect to server fqdn.com:902':
We identified that VMs located on 3 esxi hosts encounter the error above.
We noticed that on affected esxi hosts nothing is listen on port 902 even when we already had enough free space on root ramdisk:
# esxcli network ip connection | grep :902
VMs which no longer encountered issue with VM console access were located on esxi hosts where 'busybox' listened on port 902:
We decided to put affected esxi hosts into MM (Maintenance Mode) and reboot. After esxi host reboot 'busybox' started to listen on port 902 and VM console issue gone.
The main take-away is that full root ramdisk condition is abnormal - we have to remember that in *nix world everything is a file it could explain why some hosts cannot create TCP socket for 902 port when root was full even after we got some free space on root ramdisk.
Here all steps in one printscreen:
The End.
We cannot vmotion VMs to another esxi hosts in cluster.
We login into esxi hosts and noticed that root Ramdisk is full:
# vdf -h | tail -6
The uptime of esxi hosts was impressive:
# uptime
When we tried to get information about Virtual Machines using vim-cmd command we got error:
# vim-cmd vmsvc/getallvms
We tried to figure out what consumed space on root in Ramdisk, we run command:
# find / -size +10k -exec du -h {} \; | egrep -v volumes | egrep -v disks | less
I spotted a lot of EMCProvider logs in /opt/emc/cim/log
# ls -l | head -5
And bingo! these logs eat the space:
# du -h /opt/emc/cim/log/
It seems that EMCProvider logs haven't rotated and fulfilled root in Ramdisk. I couldn't find any parameter in conf file to setup rotation of EMCProvider logs - it is more feature than bug ;)
We deleted logs older than 200 days (eventually we deleted all EMCProvider logs older than 1 day) on esxi hosts in cluster:
# cd /opt/emc/cim/log/
# find . -name '*.log' -mtime +200 -exec rm -f {} \;
We got some free space on root and were able to got access to some VM console, but some VMs started to show another error 'Unable to connect to the MKS: Failed to connect to server fqdn.com:902':
We identified that VMs located on 3 esxi hosts encounter the error above.
We noticed that on affected esxi hosts nothing is listen on port 902 even when we already had enough free space on root ramdisk:
# esxcli network ip connection | grep :902
VMs which no longer encountered issue with VM console access were located on esxi hosts where 'busybox' listened on port 902:
We decided to put affected esxi hosts into MM (Maintenance Mode) and reboot. After esxi host reboot 'busybox' started to listen on port 902 and VM console issue gone.
The main take-away is that full root ramdisk condition is abnormal - we have to remember that in *nix world everything is a file it could explain why some hosts cannot create TCP socket for 902 port when root was full even after we got some free space on root ramdisk.
Here all steps in one printscreen:
The End.
Saturday, 28 June 2014
Incompatible repository. Provide a repository from vSphere Update Manager Download Service 5.x.y
I tried to configure options: 'Use a shared repository' providing full path to catalog but clicking 'Validate URL' I get error:
It is important to answer question what 'Validate URL' button is checking ?
Only path to catalog? No.
Path to patch repository with valid patch metadata? Yes.
If the patch catalog e.g. E:\Data\ is empty you will get error 'Invalid repository' because the content of folder is empty, no patch metadata.
According to VMware Documentation 5.1: http://bit.ly/1iPaZsL
"UMDS 5.1 is compatible and can work with Update Manager 5.0, Update Manager 5.0 update releases and Update Manager 5.1."
According to VMware Documentation 5.5: http://bit.ly/1mqkw4c
"UMDS 5.5 is compatible and can work with Update Manager 5.0, Update Manager 5.1, Update Manager 5.5 and their respective update releases. "
What if my VUM server is in DMZ without internet access and I would like to have automated patch download?
We have 3 options:
1.) My DMZ is 'paranoid' DMZ with 'air gap' to hosts with internet access in this case you have to use portable media like USB drive or DVD/CDROM - it is not automated process :)
2.) My DMZ servers can have http/https access to UMDS server it means that security department is happy to open port 80 or 443 - in such case you can follow e.g. http://bit.ly/1vh66sy
3.) My DMZ servers can communicate via UNC shares with hosts with internet access.It means that your security department is happy to open SMB over TCP port 445 (for Windows 2008). You can follow VMware KB http://kb.vmware.com/kb/1000627 or you can follow procedure below.
I don't have to add that security department always loves option 1 :)
Prerequisites: VMware Update Manager was installed in DMZ Server A, VMware UMDS service was installed on Server B with internet access.
1.) Log in to Server A (with VUM)
2.) Setting up shared folder on VMware Update Manager Server A
a.) Log in to each VMware Update Manager server that you will be installing that does not have an Internet connection as an Administrator.
b.) Create a shared folder that will have the exported patches from the Update Manager Download Service server at the desired drive location by going to My Computer > <Drive Letter>. For example, E:\.
c.) Right-click the My Computer window and click New > Folder and name it Data.
d.) Right-click the newly create Data folder and click Properties.
e.) Click the Sharing tab, and then click Advanced Sharing.
f.) Click Share this folder.
g.) Change the default share name to Repository.
h.) Click Permissions > Add > Locations and change location to the local computer.
i.) Under Enter the object names to select, type Administrators and then click Check Names.
j.) Click OK to save changes.
k.) Click Administrators.
l.) Click the Allow check box to give Administrators Full Control.
m.) Click OK to save changes.
3.) Log in to Server B (with UMDS)
a.) Map to the network share(s) that were created by navigating to My Computer > Map Network Drive.
b.) Choose a drive letter. For example, U:\.
c.) Click Browse and navigate to the VMware Update Manager server.
d.) Choose the share Repository. Repeat this step for each VMware Update Manager server that does not have an Internet connection and assign it a unique drive letter.
e.) Click Reconnect at logon and then click Finish.
f.) Open CMD
g.) Run: "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -G
We see that I setup only downloading host patches for esxi5.1 and esxi5.5 and I disabled downloading Virtual Appliances upgrades.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --disable-host --disable-va
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S -e embeddedEsx-5.1.0-INTL embeddedEsx-5.5.0-INTL
h.) Change patch store for UMDS:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --patch-store U:\
i.) If we decided using default patch store (and our export store is U:\) we have to export patches:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -E --export-store U:\
j.) The patch store and export store cannot be configured to the same path.
k.) We start downloading patches to our patch store.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -D
4.) As soon as metadata is downloaded to patch store on Server B (UMDS)we can validate successfully Shared repository on Server A (VUM)
the end.
It is important to answer question what 'Validate URL' button is checking ?
Only path to catalog? No.
Path to patch repository with valid patch metadata? Yes.
If the patch catalog e.g. E:\Data\ is empty you will get error 'Invalid repository' because the content of folder is empty, no patch metadata.
According to VMware Documentation 5.1: http://bit.ly/1iPaZsL
"UMDS 5.1 is compatible and can work with Update Manager 5.0, Update Manager 5.0 update releases and Update Manager 5.1."
According to VMware Documentation 5.5: http://bit.ly/1mqkw4c
"UMDS 5.5 is compatible and can work with Update Manager 5.0, Update Manager 5.1, Update Manager 5.5 and their respective update releases. "
What if my VUM server is in DMZ without internet access and I would like to have automated patch download?
We have 3 options:
1.) My DMZ is 'paranoid' DMZ with 'air gap' to hosts with internet access in this case you have to use portable media like USB drive or DVD/CDROM - it is not automated process :)
2.) My DMZ servers can have http/https access to UMDS server it means that security department is happy to open port 80 or 443 - in such case you can follow e.g. http://bit.ly/1vh66sy
3.) My DMZ servers can communicate via UNC shares with hosts with internet access.It means that your security department is happy to open SMB over TCP port 445 (for Windows 2008). You can follow VMware KB http://kb.vmware.com/kb/1000627 or you can follow procedure below.
I don't have to add that security department always loves option 1 :)
Prerequisites: VMware Update Manager was installed in DMZ Server A, VMware UMDS service was installed on Server B with internet access.
1.) Log in to Server A (with VUM)
2.) Setting up shared folder on VMware Update Manager Server A
a.) Log in to each VMware Update Manager server that you will be installing that does not have an Internet connection as an Administrator.
b.) Create a shared folder that will have the exported patches from the Update Manager Download Service server at the desired drive location by going to My Computer > <Drive Letter>. For example, E:\.
c.) Right-click the My Computer window and click New > Folder and name it Data.
d.) Right-click the newly create Data folder and click Properties.
e.) Click the Sharing tab, and then click Advanced Sharing.
f.) Click Share this folder.
g.) Change the default share name to Repository.
h.) Click Permissions > Add > Locations and change location to the local computer.
i.) Under Enter the object names to select, type Administrators and then click Check Names.
j.) Click OK to save changes.
k.) Click Administrators.
l.) Click the Allow check box to give Administrators Full Control.
m.) Click OK to save changes.
3.) Log in to Server B (with UMDS)
a.) Map to the network share(s) that were created by navigating to My Computer > Map Network Drive.
b.) Choose a drive letter. For example, U:\.
c.) Click Browse and navigate to the VMware Update Manager server.
d.) Choose the share Repository. Repeat this step for each VMware Update Manager server that does not have an Internet connection and assign it a unique drive letter.
e.) Click Reconnect at logon and then click Finish.
f.) Open CMD
g.) Run: "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -G
We see that I setup only downloading host patches for esxi5.1 and esxi5.5 and I disabled downloading Virtual Appliances upgrades.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --disable-host --disable-va
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S -e embeddedEsx-5.1.0-INTL embeddedEsx-5.5.0-INTL
h.) Change patch store for UMDS:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --patch-store U:\
i.) If we decided using default patch store (and our export store is U:\) we have to export patches:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -E --export-store U:\
j.) The patch store and export store cannot be configured to the same path.
k.) We start downloading patches to our patch store.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -D
4.) As soon as metadata is downloaded to patch store on Server B (UMDS)we can validate successfully Shared repository on Server A (VUM)
the end.
Tuesday, 24 June 2014
EMC XtremeIO with VMware View very high CPU usage on ESXi hosts
New vBlock for extreme applications, 4 Cisco B200 M3 blades, 1000 VMs are deployed - CPU on all blades hit the roof. We didn't expect that.
In XtremIO logs we notice a huge amount of SCSI reservation logs:
<info>2014-06-11 15:09:59.507783 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:5f, vol=12, opcode=42
<info>2014-06-11 15:09:59.705085 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=12, opcode=42
<info>2014-06-11 15:10:00.314514 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=6, opcode=42
<info>2014-06-11 15:10:00.973382 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=42
<info>2014-06-11 15:10:00.973393 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 15:10:01.088004 localhost xtremapp: [8474(9775 nb_truck_5)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=10, opcode=42
<info>2014-06-11 15:13:53.936724 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 16:24:34.787011 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=16, opcode
Checking VAAI settings and we see human-error:
~ # esxcfg-info -o | egrep -B 8 '/HardwareAcceleratedMove|/HardwareAcceleratedLocking'
\==+Advanced Integer Option :
|----Option Name.....................................HardwareAcceleratedLocking
|----Current Value...................................0
|----Default Value...................................1
|----Min Value.......................................0
|----Max Value.......................................1
|----Hidden..........................................false
|----Parent........................................../VMFS3/
|----Path............................................/VMFS3/HardwareAcceleratedLocking
--
\==+Advanced Integer Option :
|----Option Name.....................................HardwareAcceleratedMove
|----Current Value...................................0
|----Default Value...................................1
|----Min Value.......................................0
|----Max Value.......................................1
|----Hidden..........................................false
|----Parent........................................../DataMover/
|----Path............................................/DataMover/HardwareAcceleratedMove
Enabling VAAI primitives:
In XtremIO logs we notice a huge amount of SCSI reservation logs:
<info>2014-06-11 15:09:59.507783 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:5f, vol=12, opcode=42
<info>2014-06-11 15:09:59.705085 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=12, opcode=42
<info>2014-06-11 15:10:00.314514 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:b1:7f, vol=6, opcode=42
<info>2014-06-11 15:10:00.973382 localhost xtremapp: [8474(9771 nb_truck_1)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=42
<info>2014-06-11 15:10:00.973393 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 15:10:01.088004 localhost xtremapp: [8474(9775 nb_truck_5)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=10, opcode=42
<info>2014-06-11 15:13:53.936724 localhost xtremapp: [8474(9774 nb_truck_4)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:4f, vol=10, opcode=40
<info>2014-06-11 16:24:34.787011 localhost xtremapp: [8474(9770 nb_truck_0)]: I/O operation failure: SCSI2 reservation conflict for 20:00:00:25:b5:02:a0:7f, vol=16, opcode
Checking VAAI settings and we see human-error:
~ # esxcfg-info -o | egrep -B 8 '/HardwareAcceleratedMove|/HardwareAcceleratedLocking'
\==+Advanced Integer Option :
|----Option Name.....................................HardwareAcceleratedLocking
|----Current Value...................................0
|----Default Value...................................1
|----Min Value.......................................0
|----Max Value.......................................1
|----Hidden..........................................false
|----Parent........................................../VMFS3/
|----Path............................................/VMFS3/HardwareAcceleratedLocking
--
\==+Advanced Integer Option :
|----Option Name.....................................HardwareAcceleratedMove
|----Current Value...................................0
|----Default Value...................................1
|----Min Value.......................................0
|----Max Value.......................................1
|----Hidden..........................................false
|----Parent........................................../DataMover/
|----Path............................................/DataMover/HardwareAcceleratedMove
Enabling VAAI primitives:
# esxcli system settings advanced set -o /DataMover/HardwareAcceleratedMove -i 1
# esxcli system settings advanced set -o /VMFS3/HardwareAcceleratedLocking -i 1
SCSI reservations disappear from XtremeIO logs.
pCPUs utilisation around 50% during deployment 1000 VMs. After ~2hrs 1000 desktops ready to use. That's really extreme!!!
Remember to get extreme performance keep VAAI primitives enabled on vSphere hosts !!!
the end.
Friday, 13 June 2014
VMware vCenter stopped - MSSQL database is full
1.) On windows machine we see that disk is full.
2.) In vpxd.log or Microsoft event log we can notice:
2014-03-30T09:16:27.494+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object 'dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)"
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] SQL execution failed: INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] Diagnostic data from driver is 42000:1:1105:[Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded.
3.) To address this issue we have to follow steps from VMware KB:
http://kb.vmware.com/kb/1025914
a.) Stop vcenter server
b.) Extend virtual disk which is full for e.g. 1GB in VMware viClient.
4.) Login to Windows machine where MS SQL is installed
5.) Start 'Microsoft SQL Server Management Studio'
6.) Choose vcenter database and expand Tables
7.) Find dbo.VPX.PARAMETER
8.) Right click on dbo.VPX.PARAMETER and choose from context menu 'Edit Top 200 Rows'
9.) Find event.maxAge, event.maxAgeEnabled parameters and change values to 30 and true.
10.)Find task.maxAge, task.maxAgeEnabled parameters and change values to 30 and true,
11.)Expand vcenter DB go to Programmability -> Stored Procedures
12.)Expand Stored Procedures right click on dbo.cleanup_events_tasks_proc
13.)From context menu choose Exacute Stored Procedure...
14.) Click OK
15.) Depending on database size the task can run long period of time
16.) Go to vCenter -> Administration -> vCenter Server Settings...
17.) Parameters event.maxAge, event.maxAgeEnabled, task.maxAge, task.maxAgeEnabled are set in Database Retention Policy in vCenter settings.
Executing this procedure you delete older tasks and events in some environments with compliance requirements e.g. PCI DSS you should take backup to keep full history of events.
end
2.) In vpxd.log or Microsoft event log we can notice:
2014-03-30T09:16:27.494+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object 'dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)"
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] SQL execution failed: INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] Diagnostic data from driver is 42000:1:1105:[Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded.
3.) To address this issue we have to follow steps from VMware KB:
http://kb.vmware.com/kb/1025914
a.) Stop vcenter server
b.) Extend virtual disk which is full for e.g. 1GB in VMware viClient.
4.) Login to Windows machine where MS SQL is installed
5.) Start 'Microsoft SQL Server Management Studio'
6.) Choose vcenter database and expand Tables
7.) Find dbo.VPX.PARAMETER
8.) Right click on dbo.VPX.PARAMETER and choose from context menu 'Edit Top 200 Rows'
9.) Find event.maxAge, event.maxAgeEnabled parameters and change values to 30 and true.
10.)Find task.maxAge, task.maxAgeEnabled parameters and change values to 30 and true,
11.)Expand vcenter DB go to Programmability -> Stored Procedures
13.)From context menu choose Exacute Stored Procedure...
14.) Click OK
15.) Depending on database size the task can run long period of time
16.) Go to vCenter -> Administration -> vCenter Server Settings...
17.) Parameters event.maxAge, event.maxAgeEnabled, task.maxAge, task.maxAgeEnabled are set in Database Retention Policy in vCenter settings.
Executing this procedure you delete older tasks and events in some environments with compliance requirements e.g. PCI DSS you should take backup to keep full history of events.
end
Sunday, 8 June 2014
Cannot configure a local host OS identity source in VMware vSphere 5.1 Single Sign-On [SSO]
I get quite regular requests that someone cannot configure local host OS identity source in VMware SSO.
First of all we have to answer question why we encounter this issue?
In VMware vCenter 5.1/5.5 Single Sign-On (SSO) we basically have three modes:
* Basic
* Primary / High Availability Cluster (HA)
* Multisite
Please see VMware KB 2035817 http://kb.vmware.com/kb/2035817
In Primary / High Availablity and Multisite Single Sign-On modes, there is no local operating system identity source.
Please see VMware Documentation: http://bit.ly/TuFNDH
1.) Check what SSO mode was installed.
a.) Press WIN + R and type regedit
b.) Go to HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware Infrastructure\SSOServer and check value fot SetupType
c.) Because real men don't click you can run in command prompt
> REG QUERY "HKLM\SOFTWARE\VMware, Inc.\VMware Infrastructure\SSOServer" | findstr /i SetupType
In this case SSO was installed in Basic standalone mode therefore I can use local OS identity source. If you see value Primary / HA or Multisite you cannot use local OS identity source
2.) What in situation I have e.g. SSO in Primary mode and do not want to use Active Directory or openLDAP as identity source (seems quite unusual in enterprise but it is still the case).
a.) If your requirements is local OS identity source to demote Primary or Multisite to Basic mode you need re-install vcenter stack.
b.) As workaround solution you can consider to add users to System-Domain and assign appropriate permissions and roles to vCenter objects.
The End.
First of all we have to answer question why we encounter this issue?
In VMware vCenter 5.1/5.5 Single Sign-On (SSO) we basically have three modes:
* Basic
* Primary / High Availability Cluster (HA)
* Multisite
Please see VMware KB 2035817 http://kb.vmware.com/kb/2035817
In Primary / High Availablity and Multisite Single Sign-On modes, there is no local operating system identity source.
Please see VMware Documentation: http://bit.ly/TuFNDH
1.) Check what SSO mode was installed.
a.) Press WIN + R and type regedit
b.) Go to HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware Infrastructure\SSOServer and check value fot SetupType
> REG QUERY "HKLM\SOFTWARE\VMware, Inc.\VMware Infrastructure\SSOServer" | findstr /i SetupType
In this case SSO was installed in Basic standalone mode therefore I can use local OS identity source. If you see value Primary / HA or Multisite you cannot use local OS identity source
2.) What in situation I have e.g. SSO in Primary mode and do not want to use Active Directory or openLDAP as identity source (seems quite unusual in enterprise but it is still the case).
a.) If your requirements is local OS identity source to demote Primary or Multisite to Basic mode you need re-install vcenter stack.
b.) As workaround solution you can consider to add users to System-Domain and assign appropriate permissions and roles to vCenter objects.
The End.
Subscribe to:
Posts (Atom)