HPE Storage Users Group

BryanW wrote:Are you using AO, AFC or both for the 8 SSDs?
Was AO running when you saw the latency?

Also the first thing I would look for when there is unexplained LD/VV latency is run the "Physical Drive Compare by Performance" report for the same period and look for a failing physical disk

Absent a failing PD, it could be a few of things - reply with what you see

When trying to run any reports at all, it just sits and spins.. Never kicks out the report??

So I'm having the issue right now.. Running some CLI commands and notice the .srdata VV pulling over 2,600 iops??

According to NinjaStars my (28) 900GB 10K drives in R5 only pull 3,112 total..

My SSD say 98,000 and the VV name for that is "VDIParentReplicas"..

So which commands are you running?

.srdata is the volume where all performance metrics are stored, so if you are looking at historical data it is expected that there is activity on .srdata and as always you want the response as quickly as possible so it will take what performance it can.

MammaGutt wrote:So which commands are you running?

.srdata is the volume where all performance metrics are stored, so if you are looking at historical data it is expected that there is activity on .srdata and as always you want the response as quickly as possible so it will take what performance it can.

Thanks for the reply..

That particular command was statvv so I can see which VV are generating the most I/O.. When I was having the slowness issue, which lasted for hours, the total I/O was averaging between 3K-5K..

MammaGutt wrote:In CLI, «statpd -rw», «statvv -rw -ni», «statvlun -ni -rw» and «statvlun -hostsum -ni -rw» should give you some pointers. If statvlun is high and statvv is low, your issue is 99% host or fabric issue. With and without hostsum can help you understand if it is volume or host related. If statvv is high, the latency is due to something internal on the 3PAR, if statpd latency is high, it is most likely a backend issue (not enough disks/SSD). Always remember that read iops (vlun and vv) is mostly read from disk (around 5-10msec) while write should hit cache (1-ish msec).

So are you saying when running these commands to focus and compare the Svt MS columns?

Thanks

MammaGutt wrote:In CLI, «statpd -rw», «statvv -rw -ni», «statvlun -ni -rw» and «statvlun -hostsum -ni -rw» should give you some pointers. If statvlun is high and statvv is low, your issue is 99% host or fabric issue. With and without hostsum can help you understand if it is volume or host related. If statvv is high, the latency is due to something internal on the 3PAR, if statpd latency is high, it is most likely a backend issue (not enough disks/SSD). Always remember that read iops (vlun and vv) is mostly read from disk (around 5-10msec) while write should hit cache (1-ish msec).

Hello Mamma.. Back at it with my problem SAN.. So we added (8) 480GB SSD.. We manually put a few things in to the newly added SSD.. Our policy is to not enable AO on any of our SANS (Please don't ask me why.. it's frustrating to even talk about lol)..

Still having issues every now and again.. One thing I noticed is that the SVT time when I run: statvlun -hostsum -ni -rw .. ALmost all my hosts are over 20ms..



When I run statvv -sortcol 2,dec -ni all the svt times are under 1ms, for the most part..


I believe you mentioned that if statvlun is high and statvv is low, that it's a host or a fabric issue..

Can you recommend what I need to look at, at this point?

Thank you.

What does statport -host -rw say?

Can you provide some info about fabric?

And to refresh my memory... 3PAR OS, host OS and did you fix path policy?

MammaGutt wrote:What does statport -host -rw say?

MammaGutt wrote:Can you provide some info about fabric?

Running iSCSI (puke).. 3Par is connected to a Cisco 4510.. Ports 1 & 2 are used on each Supervisor card, One to the SAN, One to the C7K, Both are Dual homed.. The switch has no other 10Gb capability and is physically maxed out of slots..


Te5/1 c7000 VC Trunks connected trunk full 10G Unknown SFP+-*
Te5/2 3Par-SAN connected 255 full 10G 10GBase-SR
Te5/3 3Par inactive 255 full auto 10GBase-SR < Not Usable
Te5/4 3Par inactive 255 full auto 10GBase-SR < Not Usable
Te6/1 c7000 VC Trunks connected trunk full 10G Unknown SFP+-*
Te6/2 3Par-SAN connected 255 full 10G 10GBase-SR
Te6/3 inactive 1 full auto No XCVR < Not Usable
Te6/4 3Par inactive 255 full auto No XCVR < Not Usable


Only 10GB Ports 1& 2 on the supervisor cards are enabled.. Ports 3& 4 are not,

In redundant mode, the Catalyst 4507R or 4510R chassis are deployed with dual Sup7-E supervisor modules in a redundant configuration. Port group 2 becomes automatically inactive on both supervisor modules when the Catalyst 4500E system detects redundant modules installed in the chassis.

MammaGutt wrote:And to refresh my memory... 3PAR OS, host OS and did you fix path policy?

3PAR OS: 3.2.2.709 (MU6)+P99,P107,P119,P131,P135,P137,P138
Host OS: ESXi 6.5.0, 8935087, ESXi, 6.5.0, 13004031, ESXi, 6.5.0, 10719125 (not sure why these aren't all the same version. Need to follow up on that..

Pathing policy is Round Robin.

Thanks

Congratulations on the iSCSI and Vmware infrastructure

Google will give you a couple of hints that you're not alone

My suggestion... try everything ... Disable delayed ack, use software iSCSI, use hardware iSCSI and see where you get the best performance. And don't be surpriced if some servers perform best with software and others with hardware if you have different adapters (even from the same vendor) ... and hardware iSCSI can sometimes run into some "interesting" performance scenarios when simply upgrading the driver.

Also be aware that many network switches are highly oversubscribed and with very small port buffers. I've never seen the C4500 series on any list of recommended iSCSI switches but with the limited bandwidth you're using it shouldn't matter a lot.