As said, disk throughput is not going up when it happens, its just disk speed seems to drop dramatically (ie, latency goes up, ie, it takes longer to process a disk request).
CPU utilisation shows nothing odd either before or during the glitches, though difficult to tell for sure, as by its nature with "several" VMs, its very spikey. Certainly at no point are any of the Xeon's cores looking stressed.
Sounds to me like the RAID controller's having to stop and scratch its' head every so often. Does it report any stats about what it's doing?
I wonder if it's getting errors on one of the devices and retrying? I had a PERC3 that would stop and think every so often. I had a couple of corrupted files, then eventually found out that every time I MD5'ed a partition on it, I got a different answer yet it reported nothing unusual! That went in the bin with immediate effect.
I did some offline Array Diagnostics (or whatever wankword HP have come up with now, I think its now branded Smart Storage Administrator, no wonder I had to do a double take to pick the right option), no reported errors on the array card, and array card was reporting the underlying disks as healthy, with all the SMART parameters well within spec.
The HP Smart Array controllers are pretty robust, although I have seen a problem where it refused to rebuild a disk, which happened on the previous Proliant we had, as you may recall. Blowing the array away and rebuilding from scratch was the solution provided by HP, which worked, along with a firmware upgrade to prevent it happening again.
Given the cost, it would be a leap of faith to replace the card based on diags so far.