For the past month or so I’ve been working on a problem a client has with his VMware farm(s).
They’re using Vizioncore vReplicator to replicate VMs from their production farm (containing three VMware ESX 3.5 servers connected to an HP EVA fibre SAN) to their backup facility (two VMware ESX 3.5 servers with a similar HP EVA SAN). All ESX servers use multiple 1 Gbit/sec network links to an HP ProCurve switch, which is connected to its siblings by 10 Gbit/sec fibre links.
What I’ve found so far is:
- VM performance is OK, average is 400 Mbit/sec, which is not optimal but acceptable on a Gbit link.
- Service Console performance is quite sub-par: the average is about 120 Mbit/sec, which is quite slow, especially because vReplicator uses the SC port(s) to access the VMs and perform the replication.
- The speed of the SC port(s) is limited by the maximum capacity of a single CPU core. This is by design and cannot be altered. As soon as we start a large file transfer over an SC port the CPU0 core shoots up to around 100% and stays there for the duration of the copy.
So far we haven’t been able to find a fitting solution for this problem, but I thought I’d share my findings here because there is not a whole lot of information out there on these performance issues. The support engineers at Vizioncore and VMware have been quite good and very helpful, but it seems that we’re running into set boundaries within the VMWare ESX Server. The Service Console is basically a VM itself, but with a fixed upper limit of CPU power available of 100% of the first CPU core. Add to that that vReplicator can only use SC ports, and you have our problem right there.
I do like working with VMware and also quite like using vReplicator, but these limits are something to keep an eye on when you’re planning to run large replications, especially if you want them to run within a certain time frame.
I’ll update here if and when we find a cure for this problem…
Posts
