Sunday 25 December 2016

VCAP-DCV :: Harden a vSphere 6.x Deployment

1.)  Enable and configure ESXi Lockdown mode (Normal)

1.1.) To check if Lockdown mode is enabled: 
[root@vesxi60-1:~] vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
false


1.2.) To disable Lockdown mode: 
vim-cmd -U dcui vimsvc/auth/lockdown_mode_exit

1.3.)To enable Lockdown mode: 
vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter

2.) We have to use the GUI to enable ESXi Strict mode




3.) Configure a user on the Lockdown Mode Exception Users list

3.1.) We can add DCUI users:

[root@vesxi60-1:~] vim-cmd hostsvc/advopt/view DCUI
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "DCUI.Access", 
      value = "root"
   }
]

[root@vesxi60-1:~] vim-cmd hostsvc/advopt/update DCUI.Access string root,kb
[root@vesxi60-1:~] vim-cmd hostsvc/advopt/view DCUI
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "DCUI.Access", 
      value = "kb, root"
   }
]

3.2.) To add exception users we need to use GUI:




4.) Customize SSH settings for increased security  

4.1.) Check if the SSH is enabled after reboot:

[root@vesxi60-1] chkconfig --list SSH
SSH                on

4.2.) Check the timeout for ssh session where 0 is infinite timeout:

[root@vesxi60-1:~] esxcfg-advcfg -q -g /UserVars/ESXiShellTimeOut
0

[root@vesxi60-1:~] esxcli system settings advanced list -o /UserVars/ESXiShellTimeOut
   Path: /UserVars/ESXiShellTimeOut
   Type: integer
   Int Value: 0
   Default Int Value: 0
   Min Value: 0
   Max Value: 86400
   String Value: 
   Default String Value: 
   Valid Characters: 

   Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables).  Takes effect after the services are restarted. 

4.3.) Setup SSH timeout to 1 hour

[root@vesxi60-1:~] esxcli system settings advanced set -o /UserVars/ESXiShellTimeOut -i 3600
[root@vesxi60-1:~] /etc/init.d/SSH restart
SSH login disabled
SSH login enabled
[root@vesxi60-1:~] esxcli system settings advanced list -o /UserVars/ESXiShellTimeOut
   Path: /UserVars/ESXiShellTimeOut
   Type: integer
   Int Value: 3600
   Default Int Value: 0
   Min Value: 0
   Max Value: 86400
   String Value: 
   Default String Value: 
   Valid Characters: 

   Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables).  Takes effect after the services are restarted. 





Wednesday 30 November 2016

VCAP-DCV :: Configure and manage Active Directory integration from CLI

Before we join the esxi host into Active Directory domain we need to check firewall settings:

1.) Enable firewall rule activeDirectoryAll :


[root@kb-e01:~] esxcli network firewall ruleset set -r activeDirectoryAll --enabled 1

[root@kb-e01:~] esxcli network firewall ruleset list
Name Enabled
------------------------ -------
sshServer true
sshClient false
..snip..snip..

webAccess true
vMotion true
vSphereClient true
activeDirectoryAll true
NFC true
...snip..snip...


2.) Change the default ESX Admins group to custom one kblnrz_esx_admins (this group needs to be created in Active Directory before esxi joined AD domain!):

[root@kb-e01:~] vim-cmd hostsvc/advopt/update "Config.HostAgent.plugins.hostsvc.esxAdminsGroup" string "kblnrz_esx_admins"

[root@kb-e01:~] vim-cmd hostsvc/advopt/view "Config.HostAgent.plugins.hostsvc.esxAdminsGroup"
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Config.HostAgent.plugins.hostsvc.esxAdminsGroup",
      value = "kblnrz_esx_admins"
   }

]

3.) Start lwsmd daemon:


[root@kb-e01] /etc/init.d/lwsmd start

Starting Likewise Service Manager [memory reservation set] [starting lsass service] Starting service dependency: netlogon
Starting service dependency: lwio
Starting service dependency: rdr
Starting service: lsass
...ok
[root@kb-e01] /etc/init.d/lwsmd status
running
4.) Enable lwsmd deamon after reboot:
[root@kb-e01] chkconfig lwsmd on

5.) Check Domain Controller accessibility:
[root@kb-e01:/usr/lib/vmware/likewise/bin] ./lw-get-dc-name kblnrz.lab
Printing LWNET_DC_INFO fields:
===============================
dwDomainControllerAddressType = 23
dwFlags = 62461
dwVersion = 5
wLMToken = 65535
wNTToken = 65535
pszDomainControllerName = kb-w01.kblnrz.lab
pszDomainControllerAddress = 10.6.4.179
pucDomainGUID(hex) = 98 19 7A 57 BF F4 78 41 9D E2 DC 3A D7 F7 F3 C9
pszNetBIOSDomainName = KBLNRZ
pszFullyQualifiedDomainName = kblnrz.lab
pszDnsForestName = kblnrz.lab
pszDCSiteName = Default-First-Site-Name
pszClientSiteName = Default-First-Site-Name
pszNetBIOSHostName = KB-W01
pszUserName = <EMPTY> 

6.)  Now join the esxi host to AD domain :
[root@kb-e01:~] cd /usr/lib/vmware/likewise/bin/
[root@kb-e01:/usr/lib/vmware/likewise/bin] ./domainjoin-cli join kblnrz.lab Administrator
Joining to AD Domain: kblnrz.lab
With Computer DNS Name: kb-e01.kblnrz.lab
Administrator@KBLNRZ.LAB's password:
SUCCESS
7.) We can see that the AD group 'kblnrz_esx_admins' was automatically added.

[root@kb-e01:~] esxcli system permission list
Principal                 Is Group  Role   Role Description  
------------------------  --------  -----  ------------------
KBLNRZ\kblnrz_esx_admins      true  Admin  Full access rights
dcui                         false  Admin  Full access rights
root                         false  Admin  Full access rights

vpxuser                      false  Admin  Full access rights

8.) If the firewall rule activeDirectoryAll is not set you can receive following error:

[root@kb-e01:/usr/lib/vmware/likewise/bin] ./domainjoin-cli --loglevel verbose join kblnrz.lab Administrator

Joining to AD Domain: kblnrz.lab
With Computer DNS Name: kb-e01.kblnrz.lab
Administrator@KBLNRZ.LAB's password:
Error: NERR_DCNotFound [code 0x00000995]

9.) Enable likewise logging : http://kb.vmware.com/kb/1026554


the end...












Saturday 26 November 2016

VCAP-DCV :: Changing password options/strength and MOB

1.) Check the Security options on ESXi host:

[root@kb-e01:~] vim-cmd hostsvc/advopt/view Security
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Security.AccountLockFailures",
      value = 10
   },
   (vim.option.OptionValue) {
      key = "Security.AccountUnlockTime",
      value = 120
   },
   (vim.option.OptionValue) {
      key = "Security.PasswordQualityControl",
      value = "retry=3 min=disabled,disabled,disabled,7,7"
   }
]

2.) Weaken the password quality control e.g. to have VMware1!

[root@kb-e01:~] vim-cmd hostsvc/advopt/update "Security.PasswordQualityControl" string "retry=3 min=8,8,8,7,6"

[root@kb-e01:~] vim-cmd hostsvc/advopt/view "Security.PasswordQualityControl"
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Security.PasswordQualityControl",
      value = "retry=3 min=8,8,8,7,6"
   }
]

3.) Disable the Managed Object Browser (MOB) to reduce attack surface. (on vSphere 6.x MOB is disabled by default)

[root@kb-e01:~] vim-cmd hostsvc/advopt/view Config.HostAgent.plugins.solo
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Config.HostAgent.plugins.solo.enableMob",
      value = false
   },
   (vim.option.OptionValue) {
      key = "Config.HostAgent.plugins.solo.webServer.enableWebscriptLauncher",
      value = true
   }
]

4.) Enable the MOB to weaken the system:

[root@kb-e01:~] vim-cmd hostsvc/advopt/update Config.HostAgent.plugins.solo.enableMob bool 1
[root@kb-e01:~] vim-cmd hostsvc/advopt/view Config.HostAgent.plugins.solo
(vim.option.OptionValue) [
   (vim.option.OptionValue) {
      key = "Config.HostAgent.plugins.solo.enableMob",
      value = true
   },
   (vim.option.OptionValue) {
      key = "Config.HostAgent.plugins.solo.webServer.enableWebscriptLauncher",
      value = true
   }
]








Monday 21 November 2016

VCAP-DCV :: Add/Edit Remove users on an ESXi host from CLI

The most effective way to add local users is command-line we need 3 steps to add user using CLI using GUI we need around 10 steps ! Real men don't click ;-)

1.) Add user kb with description admin
[root@localhost:~] esxcli system account add -d "admin" -i kb -p -c
Enter value for 'password':
Enter value for 'password-confirmation':

2.) List newly created users:
[root@localhost:~] esxcli system account list
User ID  Description                            
-------  -------------------------------------------
root     Administrator                          
dcui     DCUI User                              
vpxuser  VMware VirtualCenter administration account
kb       admin

3.) List permissions:
[root@localhost:~] esxcli system permission list
Principal  Is Group  Role   Role Description
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights

4.) Assign new permissions to newly created user kb:
[root@localhost:~] esxcli system permission set -i kb -r Admin

5.) List assigned role / permissions:
[root@localhost:~] esxcli system permission list
Principal  Is Group  Role   Role Description  
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
kb            false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights

6.) Remove the account
[root@esxi01:~] esxcli system account remove -i kb


UNSUPPORTED way to add user with root priviliges from command-line:

1.) Enter into /bin directory:
[root@localhost:~] cd /bin/

2.) Create symbolic link:
[root@localhost:/bin] ln -s /usr/lib/vmware/busybox/bin/busybox adduser

3.) Create new user using adduser command:
[root@localhost:/bin] adduser -s /bin/ash -g admin -G root -h / kb
Changing password for kb

You can now choose the new password.

A valid password should be a mix of upper and lower case letters,
digits, and other characters.  You can use a 7 character long
password with characters from at least 3 of these 4 classes.
An upper case letter that begins the password and a digit that
ends it do not count towards the number of character classes used.

Alternatively, if noone else can see your terminal now, you can
pick this as your password: "cvyr!Terrpr!zbzrag".

Enter new password:
Re-type new password:

4.) List newly created user:
[root@localhost:/bin] esxcli system account list
User ID  Description                            
-------  -------------------------------------------
root     Administrator                          
dcui     DCUI User                              
vpxuser  VMware VirtualCenter administration account
kb       admin

5.) Don't assign the role for this user you will lost access over SSH.
[root@esxi01:/bin] esxcli system permission list
Principal  Is Group  Role   Role Description
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights


We can only login via SSH using this method, the GUI will not allow to login because of permission. Assigning the Admin role will cause that access to SSH will be lost as well. I am not sure this is bug or feature.






Saturday 19 November 2016

VCAP-DCV :: Add/Edit Remove users on an ESXi host from GUI

How to create new user and add permissions on standalone ESXi 6.x host using C# viClient or Flings ESXi Embedded Host Client native HTLM5

Using classic C# viClient:

1.) Login to ESXi with viClient using root credentials.
2.) Click on Users tab in right pane:



3.) Right click in pane:


4.) Type Login and password and click OK:


5.) The new use kb was created:


6.) Now we have to assign Permissions for newly created user: 


7.) Right click on the pane and choose Add Permission...


8.) Click Add.. to add the user:


9.) Select the user, click Add and click OK:


10.) Assign the role from context menu in right pane:


11.) Click OK after assigning role: 


12.) User kb with role Administrator was added after refresh:


Using Flings ESXi Embedded Host Client native HTLM5

1.) Login to ESXi host using root credentials:


2.) Click Manage and Users in right pane and press Add user button:


3.) Type user and password and press Add button: 


4.) The new user kb was created: 


5.) Now we have to assign the role / permissions. Click Host in left pane: 


6.) Press Actions button and choose Permissions from context menu:


7.) Press Add user button:


8.) Select user:


9.) Select role:


10.) Press Add user botton:


11.) The role Administrator was assigned to user kb:


the end.







Monday 14 March 2016

How to map ESXi vmdk to scsi devices in Linux (RHEL, CentOS etc.) GuestOS

If you have some big VM with more than 1 virtual scsi controller and many virtual disks (usually this VM is critical e.g DB) and you have to maintenance some vmdk you have to know the mapping to GuestOS Linux disk to remove proper one.

The often use tool to map vmdk to Linux disk is lsscsi.But if in meantime you changed the scsi controler type the order of scsi device in Linux changed too and you can't rely on lsscsi output any more.


The most reliable way to find the ESXi mapping to scsi devices in Linux is using pciSlotNumber (ESXi) and Physical Slot (Linux).

I am a RPM based Linux user (Centos, RHEL, Fedora etc.)

Install packages:

# yum -y install pciutils lshw

Check businfo on Linux:

# lshw -businfo | grep scsi

pci@0000:00:10.0  scsi0        storage    53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
scsi@0:0.0.0      /dev/sda     disk       42GB SCSI Disk
scsi@0:0.0.0,1                 volume     500MiB Linux filesystem partition
scsi@0:0.0.0,2    /dev/sda2    volume     39GiB Linux LVM Physical Volume partition
scsi@0:0.1.0      /dev/sdb     disk       1073MB SCSI Disk
pci@0000:02:01.0  scsi3        storage    53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
scsi@3:0.0.0      /dev/sdc     disk       1073MB SCSI Disk
pci@0000:0b:00.0  scsi4        storage    PVSCSI SCSI Controller
scsi@4:0.0.0      /dev/sdd     disk       1073MB SCSI Disk
scsi@4:0.1.0      /dev/sdg     disk       1073MB SCSI Disk
pci@0000:13:00.0  scsi5        storage    PVSCSI SCSI Controller
scsi@5:0.0.0      /dev/sde     disk       1073MB SCSI Disk
scsi@5:0.1.0      /dev/sdf     disk       1073MB SCSI Disk
                  scsi2        storage   
scsi@2:0.0.0      /dev/cdrom   disk       DVD-RAM writer
 


Check the PCI Slot on Linux box:

# lspci -v | grep -A 2 "13:00.0" | grep Phy
    Physical Slot: 224


Login to ESXi host:

# grep -i scsi[0-9].pciSlotNumber /vmfs/volumes/a*/kb-c02/*.vmx
 
/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi0.pciSlotNumber = "16"
/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi1.pciSlotNumber = "33"
/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi3.pciSlotNumber = "192"
/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi2.pciSlotNumber = "224"

Based on that information we see that vmdk disks:

# grep -i scsi2:[0-9]*.file* /vmfs/volumes/a*/kb-c02/*.vmx
 

/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi2:0.fileName = "kb-c02_4.vmdk"
/vmfs/volumes/a0754ea2-4f1bb7f5/kb-c02/kb-c02.vmx:scsi2:1.fileName = "kb-c02_5.vmdk"
 

are mapped to:

pci@0000:13:00.0  scsi5        storage    PVSCSI SCSI Controller
scsi@5:0.0.0      /dev/sde     disk       1073MB SCSI Disk
scsi@5:0.1.0      /dev/sdf     disk       1073MB SCSI Disk



WARNING: PCI slot number will  change when you change the SCSI controller type. It means that your VM will not boot in same corner case scenario see below. PCI slot numbers are increased in area of SCSI controller type. The lowest numbers has LSI Logic Parallel.(I didn't check BusLogic Parallel which is supported only with 32bit OSes)

Let say our VM is configured as follow is bootable and up and running:



VM config file looks like that:

# grep -i scsi[0-9].pciSlotNumber /vmfs/volumes/a*/kb-c02/*.vmx
 
scsi0.pciSlotNumber = "16"
scsi1.pciSlotNumber = "33"
scsi3.pciSlotNumber = "34"
scsi2.pciSlotNumber = "192"


# grep -i scsi[0-9].virtu /vmfs/volumes/a*/kb-c02/*.vmx

 
scsi1.virtualDev = "lsilogic"
scsi3.virtualDev = "lsilogic"
scsi2.virtualDev = "pvscsi"
scsi0.virtualDev = "lsilogic"


Someone decided to change the SCSI controller type to 'LSI Logic SAS'




VM config now looks like that and VM cannot boot now the scsi controller order changed this is not Linux uniq on Windows OS we will see the same behaviour.

# grep -i scsi[0-9].pciSlotNumber /vmfs/volumes/a*/kb-c02/*.vmx
 
scsi0.pciSlotNumber = "224"
scsi1.pciSlotNumber = "33"
scsi3.pciSlotNumber = "34"
scsi2.pciSlotNumber = "192"


# grep -i scsi[0-9].virtu /vmfs/volumes/a*/kb-c02/*.vmx

 
scsi1.virtualDev = "lsilogic"
scsi3.virtualDev = "lsilogic"
scsi2.virtualDev = "pvscsi"
scsi0.virtualDev = "lsisas1068"


VMware introduced in VM Hardware version 11 new parameters in .vmx file scsiX.sasWWID (I haven't found it in previous VM Hardware versions):

# grep -i scsi[0-9].sas* /vmfs/volumes/a*/kb-c02/*.vmx
 
scsi2.sasWWID = "50 05 05 62 80 46 0d 80"
scsi3.sasWWID = "50 05 05 62 80 46 0c 80"
scsi0.sasWWID = "50 05 05 62 80 46 0f 80"

This parameter is presented to PVSCSI capabilities. Unfortunately is not presented in LSI controlers:

lspci -v | grep -A 12 "0b:00.0"
 
0b:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
    Subsystem: VMware PVSCSI SCSI Controller
    Physical Slot: 192
    Flags: bus master, fast devsel, latency 0, IRQ 19
    I/O ports at 5000 [size=8]
    Memory at fd3f8000 (64-bit, non-prefetchable) [size=32K]
    [virtual] Expansion ROM at fd300000 [disabled] [size=64K]
    Capabilities: [40] Express Endpoint, MSI 00
    Capabilities: [7c] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [94] Power Management version 3
    Capabilities: [9c] MSI-X: Enable+ Count=24 Masked-
    Capabilities: [100] Device Serial Number 80-46-0d-80-50-05-05-62


lspci -v | grep -A 12 "02:01.0"
 
02:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
    Subsystem: VMware LSI Logic Parallel SCSI Controller
    Physical Slot: 33
    Flags: bus master, medium devsel, latency 64, IRQ 19
    I/O ports at 2000 [size=256]
    Memory at fd5c0000 (64-bit, non-prefetchable) [size=128K]
    Memory at fd5e0000 (64-bit, non-prefetchable) [size=128K]
    [virtual] Expansion ROM at fd500000 [disabled] [size=16K]
    Capabilities: [f8] PCI Advanced Features
    Kernel driver in use: mptspi




Another corner scenario is when we start with LSI Logic SAS controller and attach 3 x PVSCSI controllers the last assigned slot is 1184 the same case will be with 3 x LSI Logic SAS.

# grep -i scsi[0-9].pciS* /vmfs/volumes/a*/kb-c03/*.vmx
 
scsi0.pciSlotNumber = "160"
scsi1.pciSlotNumber = "224"
scsi2.pciSlotNumber = "256"
scsi3.pciSlotNumber = "1184"


# grep -i scsi[0-9].virt* /vmfs/volumes/a*/kb-c03/*.vmx

scsi0.virtualDev = "lsisas1068"
scsi1.virtualDev = "pvscsi"
scsi2.virtualDev = "pvscsi"
scsi3.virtualDev = "pvscsi"


or 

# grep -i scsi[0-9].virt* /vmfs/volumes/a*/kb-c03/*.vmx
 
scsi0.virtualDev = "lsisas1068"
scsi1.virtualDev = "lsisas1068"
scsi2.virtualDev = "lsisas1068"
scsi3.virtualDev = "lsisas1068"


The pciSlotNumber = "1184" will be translated to Physical Slot: 161


# lspci -v | grep -A 2 "04:00.0"
 
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)
    Subsystem: VMware SAS Controller
    Physical Slot: 161


# lspci -v | grep -A 2 "04:00.0"
 
04:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
    Subsystem: VMware PVSCSI SCSI Controller
    Physical Slot: 161


WARNING: Even the pciSlotNumber is high like 1184 the Linux will see that like Physical Slot 161 it mean your linux drives will be enumerated and the last controller will show as second in Linux !
I am not sure this is a bug or feature I checked that on VMware ESXi 6.0.0 build-3029758.