3PAR 7200 poor performance

[risko]

Hi,

we have 3PAR 7200 with 12 300GB 15k FC disks and 24 2TB 7.2k NL disks.
FC disks are currently RAID5 5+1 128kB, NL disks are RAID6 6+2 128kb.
SW is 3.1.2 MU3 (was also 3.1.2)
4 array FC ports are connected to 8 Gbps FC switch ports (2 FC switches, 2 ports on each).
We use 4 VMware ESXi 5.5 hosts (each has 4 8Gbps FC HBAs connected to 2 FC switches). We also tried ESXi 5.1U1.

Even if there is no other load than our test on the 3PAR 7200, we get miserable performance.
Doesn't really matter if we use even RAID0 or different stripe size.

Test consists of moving of 1 vmdk disk (50GB or 1TB) from 1 LUN to/from another LUN (FC disks LUN to NL disks LUN) or to/from LUN on another FC storage array.

We get throughput cca 300MBps. We expect something near 1000MBps with this disk configuration.
With the same test on 3 years older HP P2000G3 with 8 600GB 10k SAS disks in RAID10 or RAID5 we get throughput cca 500MBps (there is also a load of 20 production VMs on the P2000G3)


With statpd command on 3PAR we noticed, that reads are total disaster. IOPS go to maximum (600 for FC disk) but IO size is only about 40kB and resulting throughput per disk 25MB. If IO size would be something like 128kB, throughput would be ok.
During writes IOPS are moderate (150 for 15k FC disk), IO size nears sometimes 128kB, but throughput is similar.

I had an open case regarding this issue with HP, but effort ended with statement, that hardware is ok.


Any ideas how to get acceptable performance from HP 3PAR 7200?
Any idea how to force maximum IO size for disk during reads/writes?


Thank you,
Richard

[skumflum]

In my experience, you never reach the full speed potential just doing Storage vMotion. Try IOmeter and investigate the performance in different block sizes. Remember to use at testfile lager than the cache.

Have you implemented best practices for 3PAR/Vmware, like setting Multipathing policy to Round Robin?

Have you checked for any port errors in the FC Switch?

I had horrible performance on our 7400 but I don’t think your issue is the same. Search this forum and you will find my case.

[risko]

Hi,

I followed best practices for 3Par/VMware. Also looked at your thread.
There are no FC errors.
FC switch is Cisco FI 6248 (2x) with latest firmware. 3Par is directly connected to it. So are the servers.
SFP+ on HP side are HP-F, on Cisco side Cisco SFP+.


I do not expect any performance magic from 3Par.
But I don't accept it can be almost 2x slower than old MSA P2000 with fewer (8:12) and slower disks (10k:15k).
(same servers, same FC switches, same tests)
Also weird read behavior doesn't make me happy.

Thank you,
Richard

[skumflum]

The shortest path is maybe to eliminate some of the variables. Is it possible for you to connect a Windows Host directly to a VV? If you get similar poor performance, you have taken VMware out of the picture.
If you can do something similar, I would have them open the case again.
In my experience, HP (and other vendors) will expedite the case to higher support level quicker if you already have narrowed the problem.

[Cleanur]

Is the performance steady state or does it fluctuate ?

As above I would take the Hypervisor out of the equation.
Use a tool where you can build a repeatable test and ramp up the queue depth like IOmeter.
Check switch ports and 3PAR front end ports for errors or excessive retries as some transceivers have been known to cause timeouts which is especially noticeable of large sequential transfers.

[JasonAntes]

Hi,



We get throughput cca 300MBps. We expect something near 1000MBps with this disk configuration.
With the same test on 3 years older HP P2000G3 with 8 600GB 10k SAS disks in RAID10 or RAID5 we get throughput cca 500MBps (there is also a load of 20 production VMs on the P2000G3)

What is your FC connection? The 300MBps is not far off from what a single 4Gbps connection will get you. As a "Just to be sure" thing, check to make sure that the VMWare pathing is set to round robin and that you have more than 1 path available. I've had them randomly reset on me back to "last used fixed".

Also, with Storage vMotion, if you are going to the same set of disks internally on the 3PAR the VMWare throughput may be a bit jaded. Is this being measured at the array (via System Reporter)? It may be that if you are going by VMWare's measure you are only seeing half the equation, which is still not what it should be. If you are staying within the 3PAR VAII should be offloading a lot of that work and so it may not show up in VMWare's measurements, not sure on that. At any rate, I'd go directly to the array and see what it is saying if you aren't already.

[risko]

Hi,

I think, there is no problem with FC path. It is 8Gbps on both sides, no errors, multiple paths, round robin.
If I try intra array transfers (FC <-> NL), it is even slower.

If I try ATTO benchmark, results are nice - up to 1600MBps for reads. But it is pure cache.
If physical disks are involved, I get only 300MBps. And IOPS for reads are at their maximums.

ATTO - 8MB transfer size, 2GB length
hp-3par-s1 cli% statvlun -ni -hostsum -iter 1
18:59:34 01/13/2014 r/w I/O per second KBytes per sec Svt ms IOSz KB
Hostname Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
ESX13 t 3239 3239 3239 1679073 1679073 1679073 6.54 6.54 518.3 518.3 18

Thank you,
Richard

[skumflum]

ATTO is not the ideal tool because it cannot create large enough testfile. You need to be sure that the test area is greater than the 24GB Cache (maybe 2x24GB?). Use IOmeter instead, you can also simulate more real world usage like % Random of read/write

[risko]

Best IOMETER results for 12 15k FC disks on my 3Par are 350MBps.
Best IOMETER results for 24 7k NL disks on my 3Par are 170MBps.
Even in RAID0.
That is really sad...
I can build PC based storage array under $1000 with much better performance.
As HP support is of no use, I will have a look for some usable storage array.

[skumflum]

Best IOMETER results for 12 15k FC disks on my 3Par are 350MBps.
Best IOMETER results for 24 7k NL disks on my 3Par are 170MBps.
Even in RAID0.
That is really sad...
I can build PC based storage array under $1000 with much better performance.
As HP support is of no use, I will have a look for some usable storage array.

To quote the late Douglas Adams: “Don’t Panic!”. 3PAR is a premium product but if you WANT to throw it out the airlock I will send you my home address.

Did you run IOMeter on a VM or a bare-metal Windows host?

If you run into a brick wall with HP I would demand the case to be escalated. Don’t accept 1. Level opinion.

However, as I wrote before build you case…. eliminate as many unknowns before calling.