VCAP-DCV :: Configuring ESXi to use central syslog server on CentOS 7

Prepare the CentOS 7 as a syslog server.

0.) SSH to CentOS 7 server and login as a root user.

1.) Open port 514 on CentOS 7.(We will use udp for syslog)

# firewall-cmd --add-port=514/udp
# firewall-cmd --add-port=514/udp --permanent

2.) Edit /etc/rsyslog.conf uncommenting following 2 lines:

# vim /etc/rsyslog.conf

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

3.) Restart rsyslogd service

# systemctl restart rsyslogd

4.) SSH to ESXi server with root credentials

5.) Check if syslog is enabled in firewall ruleset

# esxcli network firewall ruleset list | grep syslog

syslog                      false

6.) If syslog is disabled run:

# esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true

# esxcli network firewall refresh

# esxcli network firewall ruleset list | grep syslog
syslog                      true

7.) Configure the central syslog server IP address

# esxcli system syslog config set --loghost='udp://172.30.0.40:514'
esxcli system syslog reload

8.) Send the message to the vmkernel log to check if it received by central syslog server:

# esxcli system syslog mark -s "VMware LOG !!"

9.) On CentOS 7 server you can check message log if the message was received.

# tailf /var/log/message

10.) To troubleshoot potential isssue you can use tcpdump tool

On ESXi run : # tcpdump-uw -vv -i vmk0 dst 172.30.0.40

On CentOS 7 run : # tcpdump -n -i eno33557248 port 514

You can monitor if the messages are sent and received.

The end.

VCAP-DCV :: Scratch partition configuration CLI

This procedure describe how to configure the Scratch partition using command line. This is fast and efficient - you need just login via SSH to your ESXi host.

1.) Login over SSH to ESXi host using e.g. Putty client or your terminal

2.) List your datastores 

# ls -l /vmfs/volumes/
total 3853
drwxr-xr-x    2 root     root             2 Jan 25 14:11 261b69b6-ffae9bb2
drwxr-xr-x    1 root     root             8 Jan  1  1970 5702fb79-60862c63-10b0-00505686d3de
drwxr-xr-t    1 root     root          2240 Feb  3 22:00 586410df-c23471a2-3d4a-0050568d2637
drwxr-xr-t    1 root     root          2240 Jan 28 14:29 586ac78b-5cf531fa-ca96-0050568d14b3
drwxr-xr-t    1 root     root          1680 Jan 30 21:35 588fb028-72e8890a-70ec-0050568d14b3
drwxr-xr-x    1 root     root             8 Jan  1  1970 9f691e7f-5aef34de-7c69-e495051b6cfd
lrwxr-xr-x    1 root     root            35 Mar  5 12:18 Datastore -> 588fb028-72e8890a-70ec-0050568d14b3
lrwxr-xr-x    1 root     root            17 Mar  5 12:18 NFS1 -> 261b69b6-ffae9bb2
lrwxr-xr-x    1 root     root            17 Mar  5 12:18 NFS2 -> a0e23f58-ef2c88ee
lrwxr-xr-x    1 root     root            17 Mar  5 12:18 NFS3 -> f4efbfa1-5ca3209a
drwxr-xr-x    4 root     root             9 Jan 26 14:07 a0e23f58-ef2c88ee
drwxr-xr-x    1 root     root             8 Jan  1  1970 bba0137d-45163325-50f6-9fce85fbd663
drwxr-xr-x    8 root     root            10 Jan 30 20:40 f4efbfa1-5ca3209a
lrwxr-xr-x    1 root     root            35 Mar  5 12:18 vmfs_lun01 -> 586410df-c23471a2-3d4a-0050568d2637
lrwxr-xr-x    1 root     root            35 Mar  5 12:18 vmfs_lun02 -> 586ac78b-5cf531fa-ca96-0050568d14b3
drwxr-xr-x    1 root     root           512 Mar  5 12:18 vsan:52825487009e5756-95ec4b21262a513d

lrwxr-xr-x    1 root     root            38 Mar  5 12:18 vsanDatastore -> vsan:52825487009e5756-95ec4b21262a513d

3.) We want to set Scratch partition on vmfs_lun01

# ls -la /vmfs/volumes/vmfs_lun01
lrwxr-xr-x    1 root     root            35 Mar  5 12:22 /vmfs/volumes/vmfs_lun01 -> 586410df-c23471a2-3d4a-0050568d2637

4.) We need to create the .locker directory on the vmfs_lun01

# mkdir /vmfs/volumes/vmfs_lun01/.locker-$(hostname)

5.) Check if the directory was created:

# ls -lad /vmfs/volumes/vmfs_lun01/.locker*
drwxr-xr-x    1 root     root           280 Mar  5 12:28 /vmfs/volumes/vmfs_lun01/.locker-vesxi60-2

6.) Check the current Scratch Partition settings (in this case there is no Scratch partition configured)

[root@vesxi60-2:~] vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "ScratchConfig.ConfiguredScratchLocation",
      value = ""
   }
]

7.) Set the new Scratch partition on vmfs_lun01

# /bin/vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/vmfs_lun01/.locker-vesxi60-2

# vim-cmd hostsvc/advopt/view ScratchConfig
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "ScratchConfig.ConfiguredScratchLocation",
      value = "/vmfs/volumes/586410df-c23471a2-3d4a-0050568d2637/.locker-vesxi60-2"
   }
]

8.) Put host into Maintenance Mode and Reboot

# esxcli system maintenanceMode get

Disabled

# esxcli system maintenanceMode set -e 1

# esxcli system shutdown reboot -r "Scratch partition set"

More methods to set scratch partition you can find in VMware KB :

http:/kb.vmware.com/kb/1033696

How to mount ISO/IMG from iDRAC or CIMC to Virtual Machine

As a sysadmin sometimes you are in situation to mount some img or iso file to VM e.g. to grab the driver or to check what is in the img file and you have to use management network and iDRAC or CIMC. 

WARNING: This is not the most efficient way to e.g. install OS inside VM but in some corner scenarios it could be useful.

Please find the step-by-step procedure how to mount the iDRAC or CIMC to VM. 

Cisco C-Series servers:

1.) Activate the Virtual Device:

2.) Virtual Device activated:

3.)Choose the image file to mount over CIMC KVM console:

4.) Press Map Device button:

5.) The image is mounted:

6.) In VM choose from context menu Edit Settings.. -> Hardware tab -> Add button -> SCSI Device -> Next

7.) In SCSi Device choose CD/DVD Cisco -> Next

8.) Click Finish

9.) In hardware tab of VM we see new SCSI Device - our ISO from CIMC -> Click OK

10.) In Windows VM we see that the ISO is mounted and available

Dell Rack servers with iDRAC:

1.) Connect Virtual Media :

2.)Map CD/DVD :

3.) Choose image to mount and Map Device

4.) Click Map Device

5.) In VM choose from context menu Edit Settings.. -> Hardware tab -> Add button -> SCSI Device -> Next

6.) Choose CD/DVD iDRAC

7.) Click Finish

8.) The new device will appear in Hardware tab

9.) Inside VM GuestOS you can see that image is available:

The end...

VCAP-DCV :: Harden a vSphere 6.x Deployment

1.)  Enable and configure ESXi Lockdown mode (Normal)

1.1.) To check if Lockdown mode is enabled: 
[root@vesxi60-1:~] vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
false

1.2.) To disable Lockdown mode: 
vim-cmd -U dcui vimsvc/auth/lockdown_mode_exit

1.3.)To enable Lockdown mode: 
vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter

2.) We have to use the GUI to enable ESXi Strict mode

3.) Configure a user on the Lockdown Mode Exception Users list

3.1.) We can add DCUI users:

[root@vesxi60-1:~] vim-cmd hostsvc/advopt/view DCUI
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "DCUI.Access", 
      value = "root"
   }
]

[root@vesxi60-1:~] vim-cmd hostsvc/advopt/update DCUI.Access string root,kb

[root@vesxi60-1:~] vim-cmd hostsvc/advopt/view DCUI

(vim.option.OptionValue) [

   (vim.option.OptionValue) {

      key = "DCUI.Access", 

      value = "kb, root"

   }

]

3.2.) To add exception users we need to use GUI:

4.) Customize SSH settings for increased security  

4.1.) Check if the SSH is enabled after reboot:

[root@vesxi60-1] chkconfig --list SSH
SSH                on

4.2.) Check the timeout for ssh session where 0 is infinite timeout:

[root@vesxi60-1:~] esxcfg-advcfg -q -g /UserVars/ESXiShellTimeOut
0

[root@vesxi60-1:~] esxcli system settings advanced list -o /UserVars/ESXiShellTimeOut
   Path: /UserVars/ESXiShellTimeOut
   Type: integer
   Int Value: 0
   Default Int Value: 0
   Min Value: 0
   Max Value: 86400
   String Value: 
   Default String Value: 
   Valid Characters: 

   Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables).  Takes effect after the services are restarted. 

4.3.) Setup SSH timeout to 1 hour

[root@vesxi60-1:~] esxcli system settings advanced set -o /UserVars/ESXiShellTimeOut -i 3600
[root@vesxi60-1:~] /etc/init.d/SSH restart
SSH login disabled
SSH login enabled
[root@vesxi60-1:~] esxcli system settings advanced list -o /UserVars/ESXiShellTimeOut
   Path: /UserVars/ESXiShellTimeOut
   Type: integer
   Int Value: 3600
   Default Int Value: 0
   Min Value: 0
   Max Value: 86400
   String Value: 
   Default String Value: 
   Valid Characters: 

   Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables).  Takes effect after the services are restarted. 

VCAP-DCV :: Configure and manage Active Directory integration from CLI

Before we join the esxi host into Active Directory domain we need to check firewall settings:

1.) Enable firewall rule activeDirectoryAll :

[root@kb-e01:~] esxcli network firewall ruleset set -r activeDirectoryAll --enabled 1

[root@kb-e01:~] esxcli network firewall ruleset list

Name Enabled

------------------------ -------

sshServer true

sshClient false

..snip..snip..

webAccess true

vMotion true

vSphereClient true

activeDirectoryAll true

NFC true

...snip..snip...

2.) Change the default ESX Admins group to custom one kblnrz_esx_admins (this group needs to be created in Active Directory before esxi joined AD domain!):

[root@kb-e01:~] vim-cmd hostsvc/advopt/update "Config.HostAgent.plugins.hostsvc.esxAdminsGroup" string "kblnrz_esx_admins"

[root@kb-e01:~] vim-cmd hostsvc/advopt/view "Config.HostAgent.plugins.hostsvc.esxAdminsGroup"

(vim.option.OptionValue) [

   (vim.option.OptionValue) {

      key = "Config.HostAgent.plugins.hostsvc.esxAdminsGroup",

      value = "kblnrz_esx_admins"

   }

]

3.) Start lwsmd daemon:

[root@kb-e01] /etc/init.d/lwsmd start

Starting Likewise Service Manager [memory reservation set] [starting lsass service] Starting service dependency: netlogon

Starting service dependency: lwio

Starting service dependency: rdr

Starting service: lsass

...ok

[root@kb-e01] /etc/init.d/lwsmd status

running

4.) Enable lwsmd deamon after reboot:

[root@kb-e01] chkconfig lwsmd on

5.) Check Domain Controller accessibility:

[root@kb-e01:/usr/lib/vmware/likewise/bin] ./lw-get-dc-name kblnrz.lab

Printing LWNET_DC_INFO fields:

===============================

dwDomainControllerAddressType = 23

dwFlags = 62461

dwVersion = 5

wLMToken = 65535

wNTToken = 65535

pszDomainControllerName = kb-w01.kblnrz.lab

pszDomainControllerAddress = 10.6.4.179

pucDomainGUID(hex) = 98 19 7A 57 BF F4 78 41 9D E2 DC 3A D7 F7 F3 C9

pszNetBIOSDomainName = KBLNRZ

pszFullyQualifiedDomainName = kblnrz.lab

pszDnsForestName = kblnrz.lab

pszDCSiteName = Default-First-Site-Name

pszClientSiteName = Default-First-Site-Name

pszNetBIOSHostName = KB-W01

pszUserName = <EMPTY> 

6.)  Now join the esxi host to AD domain :

[root@kb-e01:~] cd /usr/lib/vmware/likewise/bin/

[root@kb-e01:/usr/lib/vmware/likewise/bin] ./domainjoin-cli join kblnrz.lab Administrator

Joining to AD Domain: kblnrz.lab

With Computer DNS Name: kb-e01.kblnrz.lab

Administrator@KBLNRZ.LAB's password:

SUCCESS

7.) We can see that the AD group 'kblnrz_esx_admins' was automatically added.

[root@kb-e01:~] esxcli system permission list
Principal                 Is Group  Role   Role Description  
------------------------  --------  -----  ------------------
KBLNRZ\kblnrz_esx_admins      true  Admin  Full access rights
dcui                         false  Admin  Full access rights
root                         false  Admin  Full access rights

vpxuser                      false  Admin  Full access rights

8.) If the firewall rule activeDirectoryAll is not set you can receive following error:

[root@kb-e01:/usr/lib/vmware/likewise/bin] ./domainjoin-cli --loglevel verbose join kblnrz.lab Administrator

Joining to AD Domain: kblnrz.lab

With Computer DNS Name: kb-e01.kblnrz.lab

Administrator@KBLNRZ.LAB's password:

Error: NERR_DCNotFound [code 0x00000995]

9.) Enable likewise logging : http://kb.vmware.com/kb/1026554

the end...

VCAP-DCV :: Changing password options/strength and MOB

1.) Check the Security options on ESXi host:

[root@kb-e01:~] vim-cmd hostsvc/advopt/view Security
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Security.AccountLockFailures",
      value = 10
   },
   (vim.option.OptionValue) {
      key = "Security.AccountUnlockTime",
      value = 120
   },
   (vim.option.OptionValue) {
      key = "Security.PasswordQualityControl",
      value = "retry=3 min=disabled,disabled,disabled,7,7"
   }
]

2.) Weaken the password quality control e.g. to have VMware1!

[root@kb-e01:~] vim-cmd hostsvc/advopt/update "Security.PasswordQualityControl" string "retry=3 min=8,8,8,7,6"

[root@kb-e01:~] vim-cmd hostsvc/advopt/view "Security.PasswordQualityControl"

(vim.option.OptionValue) [

   (vim.option.OptionValue) {

      key = "Security.PasswordQualityControl",

      value = "retry=3 min=8,8,8,7,6"

   }

]

3.) Disable the Managed Object Browser (MOB) to reduce attack surface. (on vSphere 6.x MOB is disabled by default)

[root@kb-e01:~] vim-cmd hostsvc/advopt/view Config.HostAgent.plugins.solo

(vim.option.OptionValue) [

   (vim.option.OptionValue) {

      key = "Config.HostAgent.plugins.solo.enableMob",

      value = false

   },

   (vim.option.OptionValue) {

      key = "Config.HostAgent.plugins.solo.webServer.enableWebscriptLauncher",

      value = true

   }

]

4.) Enable the MOB to weaken the system:

[root@kb-e01:~] vim-cmd hostsvc/advopt/update Config.HostAgent.plugins.solo.enableMob bool 1

[root@kb-e01:~] vim-cmd hostsvc/advopt/view Config.HostAgent.plugins.solo

(vim.option.OptionValue) [

   (vim.option.OptionValue) {

      key = "Config.HostAgent.plugins.solo.enableMob",

      value = true

   },

   (vim.option.OptionValue) {

      key = "Config.HostAgent.plugins.solo.webServer.enableWebscriptLauncher",

      value = true

   }

]

VCAP-DCV :: Add/Edit Remove users on an ESXi host from CLI

The most effective way to add local users is command-line we need 3 steps to add user using CLI using GUI we need around 10 steps ! Real men don't click ;-)

1.) Add user kb with description admin
[root@localhost:~] esxcli system account add -d "admin" -i kb -p -c
Enter value for 'password':
Enter value for 'password-confirmation':

2.) List newly created users:

[root@localhost:~] esxcli system account list

User ID  Description                            
-------  -------------------------------------------
root     Administrator                          
dcui     DCUI User                              
vpxuser  VMware VirtualCenter administration account
kb       admin

3.) List permissions:
[root@localhost:~] esxcli system permission list
Principal  Is Group  Role   Role Description
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights

4.) Assign new permissions to newly created user kb:

[root@localhost:~] esxcli system permission set -i kb -r Admin

5.) List assigned role / permissions:

[root@localhost:~] esxcli system permission list

Principal  Is Group  Role   Role Description  

---------  --------  -----  ------------------

dcui          false  Admin  Full access rights

kb            false  Admin  Full access rights

root          false  Admin  Full access rights

vpxuser       false  Admin  Full access rights

6.) Remove the account
[root@esxi01:~] esxcli system account remove -i kb

UNSUPPORTED way to add user with root priviliges from command-line:

1.) Enter into /bin directory:
[root@localhost:~] cd /bin/

2.) Create symbolic link:
[root@localhost:/bin] ln -s /usr/lib/vmware/busybox/bin/busybox adduser

3.) Create new user using adduser command:
[root@localhost:/bin] adduser -s /bin/ash -g admin -G root -h / kb
Changing password for kb

You can now choose the new password.

A valid password should be a mix of upper and lower case letters,
digits, and other characters.  You can use a 7 character long
password with characters from at least 3 of these 4 classes.
An upper case letter that begins the password and a digit that
ends it do not count towards the number of character classes used.

Alternatively, if noone else can see your terminal now, you can
pick this as your password: "cvyr!Terrpr!zbzrag".

Enter new password:
Re-type new password:

4.) List newly created user:
[root@localhost:/bin] esxcli system account list
User ID  Description                            
-------  -------------------------------------------
root     Administrator                          
dcui     DCUI User                              
vpxuser  VMware VirtualCenter administration account
kb       admin

5.) Don't assign the role for this user you will lost access over SSH.
[root@esxi01:/bin] esxcli system permission list
Principal  Is Group  Role   Role Description
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights

We can only login via SSH using this method, the GUI will not allow to login because of permission. Assigning the Admin role will cause that access to SSH will be lost as well. I am not sure this is bug or feature.