Saturday 26 April 2014

VMware vCenter cannot start after DB full log transaction disk and truncating

1.) cannot start vpxd service because  transaction log for database 'vcenter' is full. 

mem> 2014-04-19T11:39:13.860+01:00 [13940 verbose 'utilvpxdVdb'] [VpxdVdb] Failed to get next sequence number: "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]The transaction log for database 'vcenter' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases" is returned when executing SQL statement "UPDATE VPX_SEQUENCE WITH (ROWLOCK) SET ID = ? WHERE NAME = ?"


2.) After truncating logs on MSSQL DB vpxd still cannot start we noticed in vpxd log we notice backtrace and issue with ADAM

2014-04-23T13:21:38.404+01:00 [04000 info 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance UUID: FFC9E2C9-BB99-4078-89D3-94CDB2FF0848
2014-04-23T13:21:38.435+01:00 [04000 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Instance entry missing services: dbe1eda1
2014-04-23T13:21:38.467+01:00 [04000 info 'vpxservicesMoServiceDirectory'] [RegisterBuiltinServiceEndpoints]: found service endpoint: qs-endpoint.xml
2014-04-23T13:21:38.482+01:00 [04000 error 'win32vpxLdap_win32'] [LDAP Client] Failed to add LDAP entry cn=4502de5d-eb82-4713-b4ba-268731c9afc0,ou=Instances,dc=virtualcenter,dc=vmware,dc=int: 0x68 (The object already exists.)
2014-04-23T13:21:38.482+01:00 [04000 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory::AddService] Error while adding service-instance entry cn=4502de5d-eb82-4713-b4ba-268731c9afc0,ou=Instances,dc=virtualcenter,dc=vmware,dc=int
2014-04-23T13:21:38.482+01:00 [04000 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Ldap error
--> Backtrace:
--> backtrace[00] rip 000000018018aa3a
--> backtrace[01] rip 0000000180102b18
--> backtrace[02] rip 0000000180103f3e
--> backtrace[03] rip 000000018008d6db
--> backtrace[04] rip 0000000000415c2c
--> backtrace[05] rip 0000000000436512
--> backtrace[06] rip 000000013fbf05fe
--> backtrace[07] rip 000000013fbea52c
--> backtrace[08] rip 000000013fbec11b
--> backtrace[09] rip 000000013fbf23b1
--> backtrace[10] rip 000000014003e8ad
--> backtrace[11] rip 000000014003c63c
--> backtrace[12] rip 000000014025cbeb
--> backtrace[13] rip 000007fefe01a82d
--> backtrace[14] rip 000000007746652d
--> backtrace[15] rip 000000007769c521
--> 
2014-04-23T13:21:38.482+01:00 [04000 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

3.) RDP to windows machine where vCenter is installed and run ADsedit.msc

a.) <WIN> + R
b.) type in box: adsiedit.msc




c.) Connect to dc=virtualcenter,dc=vmware,dc=int 



d.) find the stale container entry:



e.) check if container is empty:




f.) take a snapshot and delete container entry:




4.) restart vpxd service

net start vpxd

5.) after truncating logs you lost performance statistics.