Here my notes from lecture of Richard Hamming:
* I am concern about the style
* Difference between whose who do and whose who don't do significant things is A MATTER OF STYLE.
* You must experience not talk !
* Try to do something more than just average person.
* You must to do the work; you have to tear what you heard and read, think about that carefully, discuss with friends and see what you can adapt yourself.
* There is not one style to be successful. You have to find style which fit you.
* Painting is good example: if you copy master style exactly you won't became great painter.
* What you made great is not appropriate for next generations
* Education is WHAT, WHEN and WHY to do things, training is HOW to do things.
* You need both theory to guide you and skills and technic to do, one without the other isn't to good.
* From Newton times to ours we double the knowledge every 17 years.
* Nowadays we see exponential growth of number of scientists
* Quick modeling - back envelop calculation - fastest way to check if someone talking non-sense on TV.
* Learning to learn - something what was learnt will be obsolete
* He spent lifetime getting background knowledge. You have to have background knowledge enough to penetrate jargon.
* Fundamentals - from the fundamentals you can drive the rest of the field. There been around for sometime and gone entirely or partly. e.g. the fundamentals of amplification of vacuum tubes doesn't count now but Nuiquist formula for gain is still valid.
* Science vs. Engineering
* Science - you should not know what you are doing. Science suppose to be exploration what you don't know.
* Engineering - you should do it if you know what you are doing.
* Nothing is pure, science involve great deal of engineering, engineering involve great deal of new material.
* Go from idle to develop items, get them on the market rapidly.
* Short term predictions are optimistic, long term predictions are pessimistic, reason is very simple - nobody believes in geometry progression.
* Predictive future is VERY VERY hard business.
* You live long enough and do enough and you end up in history books ;-)
* History is a bonk - history is rarely reported correctly. Several witness report different things, no reliable report.
* Individuals do matter
* The past is less determined than historian like to make, and the future is less open-ended as you like to believe.
* The future get great possibilities for you.
* Unforeseen inventions can ruin everything e.g. transistor killed vacuum tubes.
* All of your life takes place in the social society which has restraints. The future technologies would be less determined about what technology ca do, than social, legal and other restrains what we can do.
* Social conventions are going to stop great things may happening.
* Drunken sailor - in N-steps typically he get square root of distance N e.g 100-steps is equal 10 distance.In another hand if there is pretty girl other-where he got a distance proportional to N.
* If I can create in you a vision of where you are headed, you may a make progress proportional to N.
* If you do not have a vision you are wobble as a drunken sailor and get very little.
* One of my major purposes to get you to form reasonable vision of what you are going to do in your future, what kind of person you are going to be.
* How can I know the future? It doesn't matter much, what goal you set? Want to go that way or that way? If you have a goal you get somewhere near if not you are drunken sailor.
* Try to achieve your goal, to make something important rather just drifting.
* It's comfortable drift through the life.
* In spite of different departments (Math, Physics, Chemistry, History etc.) there is essential unity of all knowledge. When you face a different problem of unknown type. You have to find the answer and knowledge is pretty homogenous then it's no longer divided into courses, departments etc.
* Knowledge is sort of homogenous body which we specialized with certain names but it's all reconnected together.
* One life to live you warn to be more then just get by.
* The struggle to success it's what makes you what you will be, there is no escape in living with yourself in old age.
* You stuck with yourself and at old age you cannot change yourself as much as you can change as you are younger be such kind of person you wish to be an old age and start now to be such kind of person.
* Take yourself on hand and manage yourself to be a person you wish to be, to achieve the goals you wish and be more articulate than just idle drifting drunken sailor.
* The unexamined life is not worth living.
The end
Monday 20 October 2014
Sunday 12 October 2014
Windows Server 2012 R2 booting from SAN - Cisco B200M3 - VNX2 - configuration
Prerequisites:
Zoning on N5K is configured, server profile on UCS is configured, VNX2 boot lun is created and added to storage group.
1.) Create new boot policy in service profile template with one path to boot lun.
2.) Download UCS related drivers from cisco.com
3.) Check what model of HBAs you are using:
4.) You can create image from fnic drivers extracted for example using Deamon Tools:
4a.) Choose MLOM folder:
5.) Add Windows Server 2012 R2 image :
6.) During boot we can see that only one path is available :
7.) According to our boot policy we first boot from CD-ROM (iso imange). Start installing Windows :
7a) We choose Windows Server 2012 R2 with GUI :
8.) We loading cisco fnic drivers :
9.) We start seeing volumes in our case 40GB. We get alert that the amount of space is smaller that 410346MB recommendation but I click Next :
10.) Windows installer started:
11.) Error message "Windows could not finish configuring the system. To attempt to resume configuration, restart the computer" pop up after installation we restarted Blade but error manifested again :
12.) I checked release note again and noticed that 32GB minimum recommendation is for Windows Server 2012 R2 Core version without GUI :
13.) I expanded boot_lun to 60GB. You can expand or recreate from scratch.
14.) I follow steps 1-9 and encounter the same warning about 410346MB even if I had 60GB space. Clicked next and successfully installed Windows Server 2012 R2.
15.) I changed boot policy to default with 4 paths in Service Profile Template.
16.) After reboot Windows login window pop up:
17.) I login as Administrator and added Multipath I/O feature:
18.) We see that our boot LUN is seen by MPIO.
I hope you find this helpful.
The End.
Zoning on N5K is configured, server profile on UCS is configured, VNX2 boot lun is created and added to storage group.
1.) Create new boot policy in service profile template with one path to boot lun.
2.) Download UCS related drivers from cisco.com
3.) Check what model of HBAs you are using:
4.) You can create image from fnic drivers extracted for example using Deamon Tools:
4a.) Choose MLOM folder:
5.) Add Windows Server 2012 R2 image :
6.) During boot we can see that only one path is available :
7.) According to our boot policy we first boot from CD-ROM (iso imange). Start installing Windows :
7a) We choose Windows Server 2012 R2 with GUI :
8.) We loading cisco fnic drivers :
9.) We start seeing volumes in our case 40GB. We get alert that the amount of space is smaller that 410346MB recommendation but I click Next :
11.) Error message "Windows could not finish configuring the system. To attempt to resume configuration, restart the computer" pop up after installation we restarted Blade but error manifested again :
12.) I checked release note again and noticed that 32GB minimum recommendation is for Windows Server 2012 R2 Core version without GUI :
14.) I follow steps 1-9 and encounter the same warning about 410346MB even if I had 60GB space. Clicked next and successfully installed Windows Server 2012 R2.
15.) I changed boot policy to default with 4 paths in Service Profile Template.
16.) After reboot Windows login window pop up:
17.) I login as Administrator and added Multipath I/O feature:
18.) We see that our boot LUN is seen by MPIO.
I hope you find this helpful.
The End.
Thursday 11 September 2014
Cloning Virtual Machine with thin provisioned vmdks is slower than cloning Virtual Machine with thick provisioned vmdks
I encountered very interesting phenomenon during cloning VMs across the cluster. Cloning of VM with thick (eager or lazy zeroed)virtual disk (vmdk) is much faster than cloning VM with thin virtual disk (vmdk). I made some research to explain this behavior.
I made some experiments for cloning using GUI and command-line vmkfstools. As far as I know they are in different code branch of esxi.
Here we see network utilizaton during thin and thick cloning they are almost the same:
Here is disk utilization and we notice some difference, 'thick' cloning utilize more disk than 'thin' cloning:
Something interesting start to happend 'thin' cloning use much more CPU cycles than 'thick cloning':
Let's check if results are consistent from command-line. To simplify experiment I clone only on one datastore and one esxi host to bypass network overhead.
I run command below to clone 30GB (with data) thin vmdk to thin vmdk:
# time vmkfstools -v3 -i /vmfs/volumes/datastore1/thin.vmdk -d thin /vmfs/volumes/datastore1/thin_clone.vmdk
It took 1m 2.17s ~ 62 seconds
I run command below to clone the same 30GB converted to thick vmdk to thick vmdk:
# time vmkfstools -v3 -i /vmfs/volumes/datastore1/thick.vmdk -d zeroedthick /vmfs/volumes/datastore1/thick_clone.vmdk
It took 26.63 seconds
That's is much faster !
In esxtop we see that disk was clone with 231MB/s READ and 234MB/s WRITE:
For thick cloning we notice 416MB/s READ and 407MB/s WRITE:
How can we explain that ?
Cloning is heavily sequential reads workload. Sequential reads like data to be laid out sequentially on spinning disks in order to minimize head movement. As thin-provisioned virtual disk (or thin-provisioned volumes) consume drive space only on the backend as data is written to them, this data tends not to be from contiguous areas, leading to the address space of the virtual disk or volume being fragmented on the array's backend. This can lead to thin-provisioned virtual disks (or volumes) performing significantly lower than thick provisioned virtual disks (or volumes) for this particular workload type.
Please bear in mind that this has nothing to do with workload inside virtual disk. Performance overhead of VM GuestOS on thin-provisioned virtual disk (vmdk) in negligible and comparable to thick-provisionend vmdk.
In another words we have sequential access versus random access.By definition random access - read or write I/O's that are distributed throughout the relevant address space; sequential access - read or write I/O's that are logically contiguous within the relevant address space.
The End.
I made some experiments for cloning using GUI and command-line vmkfstools. As far as I know they are in different code branch of esxi.
Here we see network utilizaton during thin and thick cloning they are almost the same:
Here is disk utilization and we notice some difference, 'thick' cloning utilize more disk than 'thin' cloning:
Something interesting start to happend 'thin' cloning use much more CPU cycles than 'thick cloning':
Let's check if results are consistent from command-line. To simplify experiment I clone only on one datastore and one esxi host to bypass network overhead.
I run command below to clone 30GB (with data) thin vmdk to thin vmdk:
# time vmkfstools -v3 -i /vmfs/volumes/datastore1/thin.vmdk -d thin /vmfs/volumes/datastore1/thin_clone.vmdk
It took 1m 2.17s ~ 62 seconds
I run command below to clone the same 30GB converted to thick vmdk to thick vmdk:
# time vmkfstools -v3 -i /vmfs/volumes/datastore1/thick.vmdk -d zeroedthick /vmfs/volumes/datastore1/thick_clone.vmdk
It took 26.63 seconds
That's is much faster !
In esxtop we see that disk was clone with 231MB/s READ and 234MB/s WRITE:
For thick cloning we notice 416MB/s READ and 407MB/s WRITE:
How can we explain that ?
Cloning is heavily sequential reads workload. Sequential reads like data to be laid out sequentially on spinning disks in order to minimize head movement. As thin-provisioned virtual disk (or thin-provisioned volumes) consume drive space only on the backend as data is written to them, this data tends not to be from contiguous areas, leading to the address space of the virtual disk or volume being fragmented on the array's backend. This can lead to thin-provisioned virtual disks (or volumes) performing significantly lower than thick provisioned virtual disks (or volumes) for this particular workload type.
Please bear in mind that this has nothing to do with workload inside virtual disk. Performance overhead of VM GuestOS on thin-provisioned virtual disk (vmdk) in negligible and comparable to thick-provisionend vmdk.
In another words we have sequential access versus random access.By definition random access - read or write I/O's that are distributed throughout the relevant address space; sequential access - read or write I/O's that are logically contiguous within the relevant address space.
The End.
Saturday 6 September 2014
Notes from presentation of Jack Ma - Alibaba Group founder.
* Nothing will happen to you if you don't act
* If you act, you may benefit from that action.
* The ignorant are not afraid.
* There were three reasons behind Alibaba success:
1.) no money
* A lot of entrepreneurs didn't make a success not because they don't have money but because they had too much money. They tried to solve problems with $$$.
* Jack believes that money could not solve all the problems. Money is just an important tactic in problem solving.Alibaba might have one of China's and maybe even the world's largest cash reserves among Internet companies.
* A company's assets are like a state's armed forces you cannot use it lightly, but if you ever need to mobilize it, you must win !!!
* Money cannot be spent recklessly.
2.) Jack didn't understand technology
* Jack doesn't understand the technology behind the Internet but it doesn't mean that he doesn't respect technology. It's a tragedy for engineers that their CEO knows nothing about technology, but they are lucky also. If CEO understands technology he will be looking over engineers shoulders every day and they won't be able to get any work done.
* Jack has a great respect for their technical team, he hasn't written even one line of code ;)
* If we cannot turn out vision into reality, it is not even worth a penny.
* Jack tested whether things were useful or not as end-user.
* They respect technology but they are afraid of it, they need it to be USEFUL.
* If you cannot use the technology - regardless of how good it is - it is USELESS.
* If people believed in technology, the technology becomes productive, innovative and can impact society.
* Jack doesn't think China or the world lacks technology, it lacks appreciation and reverence for it.
3.) they never planned ;)
* Jack never have written a business plan. They wrote 'one' to seek founder.
* In 1996 nobody knows what happens.
* Plan is only useful if you execute your plan !
* Our life is a plan that is slowly unfolding.
* Change is the best plan, but you don't want to lose your own sense of direction.
* Other people in Alibaba have very good plans !!!
* Someone has to plan ;)
* We are living in a time of constant change. We have to adapt to an ever-evolving world.
* Change is what give young people opportunities.
Luck is very important. You can't wait for luck, because it won't come.
Luck is like seeds that have to be sown. Luck and wealth are the same. Sow the seeds of good luck in other people's yards.That will likely bring you more luck.
The world is changing very fast and it's hard to gauge consumer's needs - focus on small business.
This is era of customization - P4P advertisement, big data.
It's hard to imagine there are 100 millions consumers daily on Taobao.
Nobody can guarantee that you will fail.
"Young people lie awake at night thinking of thousand paths, and get up in the morning to walk the same path."
The entire Jack's story sounds familiar to Allegro Group, internet company in Poland which is a Internet giant and leader in Eastern Europe.
The End.
* If you act, you may benefit from that action.
* The ignorant are not afraid.
* There were three reasons behind Alibaba success:
1.) no money
* A lot of entrepreneurs didn't make a success not because they don't have money but because they had too much money. They tried to solve problems with $$$.
* Jack believes that money could not solve all the problems. Money is just an important tactic in problem solving.Alibaba might have one of China's and maybe even the world's largest cash reserves among Internet companies.
* A company's assets are like a state's armed forces you cannot use it lightly, but if you ever need to mobilize it, you must win !!!
* Money cannot be spent recklessly.
2.) Jack didn't understand technology
* Jack doesn't understand the technology behind the Internet but it doesn't mean that he doesn't respect technology. It's a tragedy for engineers that their CEO knows nothing about technology, but they are lucky also. If CEO understands technology he will be looking over engineers shoulders every day and they won't be able to get any work done.
* Jack has a great respect for their technical team, he hasn't written even one line of code ;)
* If we cannot turn out vision into reality, it is not even worth a penny.
* Jack tested whether things were useful or not as end-user.
* They respect technology but they are afraid of it, they need it to be USEFUL.
* If you cannot use the technology - regardless of how good it is - it is USELESS.
* If people believed in technology, the technology becomes productive, innovative and can impact society.
* Jack doesn't think China or the world lacks technology, it lacks appreciation and reverence for it.
3.) they never planned ;)
* Jack never have written a business plan. They wrote 'one' to seek founder.
* In 1996 nobody knows what happens.
* Plan is only useful if you execute your plan !
* Our life is a plan that is slowly unfolding.
* Change is the best plan, but you don't want to lose your own sense of direction.
* Other people in Alibaba have very good plans !!!
* Someone has to plan ;)
* We are living in a time of constant change. We have to adapt to an ever-evolving world.
* Change is what give young people opportunities.
Luck is very important. You can't wait for luck, because it won't come.
Luck is like seeds that have to be sown. Luck and wealth are the same. Sow the seeds of good luck in other people's yards.That will likely bring you more luck.
The world is changing very fast and it's hard to gauge consumer's needs - focus on small business.
This is era of customization - P4P advertisement, big data.
It's hard to imagine there are 100 millions consumers daily on Taobao.
Nobody can guarantee that you will fail.
"Young people lie awake at night thinking of thousand paths, and get up in the morning to walk the same path."
The entire Jack's story sounds familiar to Allegro Group, internet company in Poland which is a Internet giant and leader in Eastern Europe.
The End.
Friday 15 August 2014
Notes from INF-STO2564 session provided by Aboubacar Diare from HP
ATS - Atomic Test&Set primitive.
SCSI2 reservation - entire volume is locked
ATS - lock only block
VMFS3: layout - resources (inodes, blocks, sub-blocks etc.)organized into clusters, each cluster has an associated lock and metadata. Clusters form a cluster groups. Cluster groups repeat to make the filesystem.
ATS Values:
* space locality - esxi hosts strive for contiguous blocks of objects they manage
* reduces storage resources contention
* larger VMFS datastores size
* higher VMs density
* reduced datastore management
ATS Caveats:
* vmotion vs. storage vMotion e.g.
1.) on esxi1 we create 10 VMs which see 500GB at the beginning of 1TB volume
2.) host esxi1 is accessing this contiguous area
3.) on esxi2 we create VM which allocate 100GB on the same 1TB volume at the end of 500GB area accessed by esxi1. Both hosts work fine without any storage contention.
4.) now, we vmotion 5 VMs to esxi2,
5.) esxi2 host get access to contiguous space previously accessed by esxi1
6.) now, this hosts start contending for the resources in different area of the disk. Space locality is disturbed.
Do vMotion only when is needed e.g. DRS recommendation. Instead of causing multiple hosts access region on the disk which other hosts will need and increasing the potential for contention of these resources. Storage vMotion more preferable than classic vMotion.
It doesn't mean don't use vmotion !!! It means think when you scale out with vSphere cluster !!!
* free capacity (filling up VMFS)- need free capacity on the datastore, esxi don't like full datastores ;)
* storage vMotion -> force esxi to find contiguous space.
UNMAP primitive:
# esxcli system settings advanced set -i 0 -o /VMFS3/EnableBlockDelete
# vmkfstools -y X (X = 1 - 100% default 60%)
VMware disabled UNMAP - only manual reclaiming posssible.
Reclaiming space => huge impact on performance. UNMPAP operation create a balloon file equal to size of space to be de-allocated. Balloon file causes many WR I/O.
Avoid X > 90% !!!
You may not get back space which you expected to get back !
Percentage is derived from free space available on VMFS datastore e.g.
2TB datastore 50% full with vmkfstools -y 50 will attempt to reclaim 0.5TB (512GB) = 50% of 1TB free.
VAAI Consideration/Hidden Benefits:
* fewer commands send to the storage array.
UNMAP command is different and depends on array implementation, it could be more command than without UNMAP primitive.
Some VAAI implementation will NOT work if datastore is not aligned:
* VMFS3 datastore to 64kB
* VMFS5 datastore to 1MB
* some arrays may reject non-aligned VAAI operations
* concurrent clone operations:
-- maybe throttle by array at some threshold (current backend processing limit)
-- limit number of concurrent clone/zero to 3 or 4.
VAAI and array replication - some performance deprecation for VAAI.
Fixed I/O path policy is NOT ALUA aware and not recommended for ALUA arrays.
Changing MPIO settings for multiple LUNs:
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp device set -P VMW_PSP_RR -d $i; done
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp psp roundrobin deviceconfig set -t iops –I 1 -d $i; done
FIXED_AP - ALUA aware
ESXi5 rolled FIXED_AP functionality into the FIXED I/O path policy. FIXED_AP only applicable to ALUA capable arrays.
Is FIXED in ESXi5 recommended for ALUA arrays?
NO ! For general usage (LUN thrashing). YES ! as a tool for quickly restoring balance in an unbalanced array LUN configuration.
Round Robin - lower IOPS is better, high IOPS value for sequential workload.
SCSI2 reservation - entire volume is locked
ATS - lock only block
VMFS3: layout - resources (inodes, blocks, sub-blocks etc.)organized into clusters, each cluster has an associated lock and metadata. Clusters form a cluster groups. Cluster groups repeat to make the filesystem.
ATS Values:
* space locality - esxi hosts strive for contiguous blocks of objects they manage
* reduces storage resources contention
* larger VMFS datastores size
* higher VMs density
* reduced datastore management
ATS Caveats:
* vmotion vs. storage vMotion e.g.
1.) on esxi1 we create 10 VMs which see 500GB at the beginning of 1TB volume
2.) host esxi1 is accessing this contiguous area
3.) on esxi2 we create VM which allocate 100GB on the same 1TB volume at the end of 500GB area accessed by esxi1. Both hosts work fine without any storage contention.
4.) now, we vmotion 5 VMs to esxi2,
5.) esxi2 host get access to contiguous space previously accessed by esxi1
6.) now, this hosts start contending for the resources in different area of the disk. Space locality is disturbed.
Do vMotion only when is needed e.g. DRS recommendation. Instead of causing multiple hosts access region on the disk which other hosts will need and increasing the potential for contention of these resources. Storage vMotion more preferable than classic vMotion.
It doesn't mean don't use vmotion !!! It means think when you scale out with vSphere cluster !!!
* free capacity (filling up VMFS)- need free capacity on the datastore, esxi don't like full datastores ;)
* storage vMotion -> force esxi to find contiguous space.
UNMAP primitive:
# esxcli system settings advanced set -i 0 -o /VMFS3/EnableBlockDelete
# vmkfstools -y X (X = 1 - 100% default 60%)
VMware disabled UNMAP - only manual reclaiming posssible.
Reclaiming space => huge impact on performance. UNMPAP operation create a balloon file equal to size of space to be de-allocated. Balloon file causes many WR I/O.
Avoid X > 90% !!!
You may not get back space which you expected to get back !
Percentage is derived from free space available on VMFS datastore e.g.
2TB datastore 50% full with vmkfstools -y 50 will attempt to reclaim 0.5TB (512GB) = 50% of 1TB free.
VAAI Consideration/Hidden Benefits:
* fewer commands send to the storage array.
UNMAP command is different and depends on array implementation, it could be more command than without UNMAP primitive.
Some VAAI implementation will NOT work if datastore is not aligned:
* VMFS3 datastore to 64kB
* VMFS5 datastore to 1MB
* some arrays may reject non-aligned VAAI operations
* concurrent clone operations:
-- maybe throttle by array at some threshold (current backend processing limit)
-- limit number of concurrent clone/zero to 3 or 4.
VAAI and array replication - some performance deprecation for VAAI.
Fixed I/O path policy is NOT ALUA aware and not recommended for ALUA arrays.
Changing MPIO settings for multiple LUNs:
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp device set -P VMW_PSP_RR -d $i; done
# for i in 'esxcli storage nmp device list | grep ^naa.6001' ; do esxcli storage nmp psp roundrobin deviceconfig set -t iops –I 1 -d $i; done
FIXED_AP - ALUA aware
ESXi5 rolled FIXED_AP functionality into the FIXED I/O path policy. FIXED_AP only applicable to ALUA capable arrays.
Is FIXED in ESXi5 recommended for ALUA arrays?
NO ! For general usage (LUN thrashing). YES ! as a tool for quickly restoring balance in an unbalanced array LUN configuration.
Round Robin - lower IOPS is better, high IOPS value for sequential workload.
Tuesday 12 August 2014
No access to VM console; Unable to connect to the MKS: A general system error occured: Internal error
All 16 hosts in cluster are up and running since long time without any issue - uptime 300+ days. On all hosts we cannot get access to VM console. Opening VM console from viClient we get error "Unable to connect to the MKS: A general system error occured: Internal error".
We cannot vmotion VMs to another esxi hosts in cluster.
We login into esxi hosts and noticed that root Ramdisk is full:
# vdf -h | tail -6
The uptime of esxi hosts was impressive:
# uptime
When we tried to get information about Virtual Machines using vim-cmd command we got error:
# vim-cmd vmsvc/getallvms
We tried to figure out what consumed space on root in Ramdisk, we run command:
# find / -size +10k -exec du -h {} \; | egrep -v volumes | egrep -v disks | less
I spotted a lot of EMCProvider logs in /opt/emc/cim/log
# ls -l | head -5
And bingo! these logs eat the space:
# du -h /opt/emc/cim/log/
It seems that EMCProvider logs haven't rotated and fulfilled root in Ramdisk. I couldn't find any parameter in conf file to setup rotation of EMCProvider logs - it is more feature than bug ;)
We deleted logs older than 200 days (eventually we deleted all EMCProvider logs older than 1 day) on esxi hosts in cluster:
# cd /opt/emc/cim/log/
# find . -name '*.log' -mtime +200 -exec rm -f {} \;
We got some free space on root and were able to got access to some VM console, but some VMs started to show another error 'Unable to connect to the MKS: Failed to connect to server fqdn.com:902':
We identified that VMs located on 3 esxi hosts encounter the error above.
We noticed that on affected esxi hosts nothing is listen on port 902 even when we already had enough free space on root ramdisk:
# esxcli network ip connection | grep :902
VMs which no longer encountered issue with VM console access were located on esxi hosts where 'busybox' listened on port 902:
We decided to put affected esxi hosts into MM (Maintenance Mode) and reboot. After esxi host reboot 'busybox' started to listen on port 902 and VM console issue gone.
The main take-away is that full root ramdisk condition is abnormal - we have to remember that in *nix world everything is a file it could explain why some hosts cannot create TCP socket for 902 port when root was full even after we got some free space on root ramdisk.
Here all steps in one printscreen:
The End.
We cannot vmotion VMs to another esxi hosts in cluster.
We login into esxi hosts and noticed that root Ramdisk is full:
# vdf -h | tail -6
The uptime of esxi hosts was impressive:
# uptime
When we tried to get information about Virtual Machines using vim-cmd command we got error:
# vim-cmd vmsvc/getallvms
We tried to figure out what consumed space on root in Ramdisk, we run command:
# find / -size +10k -exec du -h {} \; | egrep -v volumes | egrep -v disks | less
I spotted a lot of EMCProvider logs in /opt/emc/cim/log
# ls -l | head -5
And bingo! these logs eat the space:
# du -h /opt/emc/cim/log/
It seems that EMCProvider logs haven't rotated and fulfilled root in Ramdisk. I couldn't find any parameter in conf file to setup rotation of EMCProvider logs - it is more feature than bug ;)
We deleted logs older than 200 days (eventually we deleted all EMCProvider logs older than 1 day) on esxi hosts in cluster:
# cd /opt/emc/cim/log/
# find . -name '*.log' -mtime +200 -exec rm -f {} \;
We got some free space on root and were able to got access to some VM console, but some VMs started to show another error 'Unable to connect to the MKS: Failed to connect to server fqdn.com:902':
We identified that VMs located on 3 esxi hosts encounter the error above.
We noticed that on affected esxi hosts nothing is listen on port 902 even when we already had enough free space on root ramdisk:
# esxcli network ip connection | grep :902
VMs which no longer encountered issue with VM console access were located on esxi hosts where 'busybox' listened on port 902:
We decided to put affected esxi hosts into MM (Maintenance Mode) and reboot. After esxi host reboot 'busybox' started to listen on port 902 and VM console issue gone.
The main take-away is that full root ramdisk condition is abnormal - we have to remember that in *nix world everything is a file it could explain why some hosts cannot create TCP socket for 902 port when root was full even after we got some free space on root ramdisk.
Here all steps in one printscreen:
The End.
Saturday 28 June 2014
Incompatible repository. Provide a repository from vSphere Update Manager Download Service 5.x.y
I tried to configure options: 'Use a shared repository' providing full path to catalog but clicking 'Validate URL' I get error:
It is important to answer question what 'Validate URL' button is checking ?
Only path to catalog? No.
Path to patch repository with valid patch metadata? Yes.
If the patch catalog e.g. E:\Data\ is empty you will get error 'Invalid repository' because the content of folder is empty, no patch metadata.
According to VMware Documentation 5.1: http://bit.ly/1iPaZsL
"UMDS 5.1 is compatible and can work with Update Manager 5.0, Update Manager 5.0 update releases and Update Manager 5.1."
According to VMware Documentation 5.5: http://bit.ly/1mqkw4c
"UMDS 5.5 is compatible and can work with Update Manager 5.0, Update Manager 5.1, Update Manager 5.5 and their respective update releases. "
What if my VUM server is in DMZ without internet access and I would like to have automated patch download?
We have 3 options:
1.) My DMZ is 'paranoid' DMZ with 'air gap' to hosts with internet access in this case you have to use portable media like USB drive or DVD/CDROM - it is not automated process :)
2.) My DMZ servers can have http/https access to UMDS server it means that security department is happy to open port 80 or 443 - in such case you can follow e.g. http://bit.ly/1vh66sy
3.) My DMZ servers can communicate via UNC shares with hosts with internet access.It means that your security department is happy to open SMB over TCP port 445 (for Windows 2008). You can follow VMware KB http://kb.vmware.com/kb/1000627 or you can follow procedure below.
I don't have to add that security department always loves option 1 :)
Prerequisites: VMware Update Manager was installed in DMZ Server A, VMware UMDS service was installed on Server B with internet access.
1.) Log in to Server A (with VUM)
2.) Setting up shared folder on VMware Update Manager Server A
a.) Log in to each VMware Update Manager server that you will be installing that does not have an Internet connection as an Administrator.
b.) Create a shared folder that will have the exported patches from the Update Manager Download Service server at the desired drive location by going to My Computer > <Drive Letter>. For example, E:\.
c.) Right-click the My Computer window and click New > Folder and name it Data.
d.) Right-click the newly create Data folder and click Properties.
e.) Click the Sharing tab, and then click Advanced Sharing.
f.) Click Share this folder.
g.) Change the default share name to Repository.
h.) Click Permissions > Add > Locations and change location to the local computer.
i.) Under Enter the object names to select, type Administrators and then click Check Names.
j.) Click OK to save changes.
k.) Click Administrators.
l.) Click the Allow check box to give Administrators Full Control.
m.) Click OK to save changes.
3.) Log in to Server B (with UMDS)
a.) Map to the network share(s) that were created by navigating to My Computer > Map Network Drive.
b.) Choose a drive letter. For example, U:\.
c.) Click Browse and navigate to the VMware Update Manager server.
d.) Choose the share Repository. Repeat this step for each VMware Update Manager server that does not have an Internet connection and assign it a unique drive letter.
e.) Click Reconnect at logon and then click Finish.
f.) Open CMD
g.) Run: "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -G
We see that I setup only downloading host patches for esxi5.1 and esxi5.5 and I disabled downloading Virtual Appliances upgrades.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --disable-host --disable-va
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S -e embeddedEsx-5.1.0-INTL embeddedEsx-5.5.0-INTL
h.) Change patch store for UMDS:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --patch-store U:\
i.) If we decided using default patch store (and our export store is U:\) we have to export patches:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -E --export-store U:\
j.) The patch store and export store cannot be configured to the same path.
k.) We start downloading patches to our patch store.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -D
4.) As soon as metadata is downloaded to patch store on Server B (UMDS)we can validate successfully Shared repository on Server A (VUM)
the end.
It is important to answer question what 'Validate URL' button is checking ?
Only path to catalog? No.
Path to patch repository with valid patch metadata? Yes.
If the patch catalog e.g. E:\Data\ is empty you will get error 'Invalid repository' because the content of folder is empty, no patch metadata.
According to VMware Documentation 5.1: http://bit.ly/1iPaZsL
"UMDS 5.1 is compatible and can work with Update Manager 5.0, Update Manager 5.0 update releases and Update Manager 5.1."
According to VMware Documentation 5.5: http://bit.ly/1mqkw4c
"UMDS 5.5 is compatible and can work with Update Manager 5.0, Update Manager 5.1, Update Manager 5.5 and their respective update releases. "
What if my VUM server is in DMZ without internet access and I would like to have automated patch download?
We have 3 options:
1.) My DMZ is 'paranoid' DMZ with 'air gap' to hosts with internet access in this case you have to use portable media like USB drive or DVD/CDROM - it is not automated process :)
2.) My DMZ servers can have http/https access to UMDS server it means that security department is happy to open port 80 or 443 - in such case you can follow e.g. http://bit.ly/1vh66sy
3.) My DMZ servers can communicate via UNC shares with hosts with internet access.It means that your security department is happy to open SMB over TCP port 445 (for Windows 2008). You can follow VMware KB http://kb.vmware.com/kb/1000627 or you can follow procedure below.
I don't have to add that security department always loves option 1 :)
Prerequisites: VMware Update Manager was installed in DMZ Server A, VMware UMDS service was installed on Server B with internet access.
1.) Log in to Server A (with VUM)
2.) Setting up shared folder on VMware Update Manager Server A
a.) Log in to each VMware Update Manager server that you will be installing that does not have an Internet connection as an Administrator.
b.) Create a shared folder that will have the exported patches from the Update Manager Download Service server at the desired drive location by going to My Computer > <Drive Letter>. For example, E:\.
c.) Right-click the My Computer window and click New > Folder and name it Data.
d.) Right-click the newly create Data folder and click Properties.
e.) Click the Sharing tab, and then click Advanced Sharing.
f.) Click Share this folder.
g.) Change the default share name to Repository.
h.) Click Permissions > Add > Locations and change location to the local computer.
i.) Under Enter the object names to select, type Administrators and then click Check Names.
j.) Click OK to save changes.
k.) Click Administrators.
l.) Click the Allow check box to give Administrators Full Control.
m.) Click OK to save changes.
3.) Log in to Server B (with UMDS)
a.) Map to the network share(s) that were created by navigating to My Computer > Map Network Drive.
b.) Choose a drive letter. For example, U:\.
c.) Click Browse and navigate to the VMware Update Manager server.
d.) Choose the share Repository. Repeat this step for each VMware Update Manager server that does not have an Internet connection and assign it a unique drive letter.
e.) Click Reconnect at logon and then click Finish.
f.) Open CMD
g.) Run: "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -G
We see that I setup only downloading host patches for esxi5.1 and esxi5.5 and I disabled downloading Virtual Appliances upgrades.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --disable-host --disable-va
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S -e embeddedEsx-5.1.0-INTL embeddedEsx-5.5.0-INTL
h.) Change patch store for UMDS:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -S --patch-store U:\
i.) If we decided using default patch store (and our export store is U:\) we have to export patches:
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -E --export-store U:\
j.) The patch store and export store cannot be configured to the same path.
k.) We start downloading patches to our patch store.
"C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe" -D
4.) As soon as metadata is downloaded to patch store on Server B (UMDS)we can validate successfully Shared repository on Server A (VUM)
the end.
Subscribe to:
Posts (Atom)