Friday 13 June 2014

VMware vCenter stopped - MSSQL database is full

1.) On windows machine we see that disk is full.

2.) In vpxd.log or Microsoft event log we can notice:

2014-03-30T09:16:27.494+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object 'dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)"
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] SQL execution failed: INSERT INTO VPX_HOST_VM_CONFIG_OPTION WITH (ROWLOCK) (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX, CONFIG_OPTION_DESC, CREATE_SUPPORTED_FLG, DEFAULT_CONFIG_OPTION_FLG) VALUES (?, ?, ?, ?, ?, ?, ?)
2014-03-30T09:16:15.853+08:00 [04120 error 'Default' opID=HB-host-87@2013030-52d23067] [VdbStatement] Diagnostic data from driver is 42000:1:1105:[Microsoft][SQL Server Native Client 10.0][SQL Server]Could not allocate space for object dbo.VPX_HOST_VM_CONFIG_OPTION'.'PK_VPX_HOST_VM_CONFIG_OPTION' in database 'vCenter' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded.


3.) To address this issue we have to follow steps from VMware KB:
http://kb.vmware.com/kb/1025914 

a.) Stop vcenter server
b.) Extend virtual disk which is full for e.g. 1GB in VMware viClient.

4.) Login to Windows machine where MS SQL is installed

5.) Start 'Microsoft SQL Server Management Studio'

6.) Choose vcenter database and expand Tables








7.) Find dbo.VPX.PARAMETER



 8.) Right click on dbo.VPX.PARAMETER and choose from context menu 'Edit Top 200 Rows'


9.) Find event.maxAge, event.maxAgeEnabled parameters and change values to 30 and true.


10.)Find task.maxAge, task.maxAgeEnabled parameters and change values to 30 and true,


11.)Expand vcenter DB go to Programmability -> Stored Procedures

 12.)Expand Stored Procedures right click on dbo.cleanup_events_tasks_proc


13.)From context menu choose Exacute Stored Procedure...



14.) Click OK


15.) Depending on database size the task can run long period of time


16.) Go to vCenter -> Administration -> vCenter Server Settings...



17.) Parameters event.maxAge, event.maxAgeEnabled, task.maxAge, task.maxAgeEnabled are set in Database Retention Policy in vCenter settings.



Executing this procedure you delete older tasks and events in some environments with compliance requirements e.g. PCI DSS you should take backup to keep full history of events.

end




3 comments:

  1. Really thanks,
    pretty clear and useful

    ReplyDelete
  2. Really thanks,
    pretty clear and useful

    ReplyDelete
  3. tanks a lot for this helpful documentation
    so clear and useful
    thank u thank u

    ReplyDelete