3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 19 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Mon Jul 02, 2018 3:56 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
BryanW wrote:
Are you using AO, AFC or both for the 8 SSDs?
Was AO running when you saw the latency?

Also the first thing I would look for when there is unexplained LD/VV latency is run the "Physical Drive Compare by Performance" report for the same period and look for a failing physical disk

Absent a failing PD, it could be a few of things - reply with what you see


When trying to run any reports at all, it just sits and spins.. Never kicks out the report??


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Mon Jul 09, 2018 3:14 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
So I'm having the issue right now.. Running some CLI commands and notice the .srdata VV pulling over 2,600 iops??

According to NinjaStars my (28) 900GB 10K drives in R5 only pull 3,112 total..

My SSD say 98,000 and the VV name for that is "VDIParentReplicas"..


Attachments:
srdata.jpg
srdata.jpg [ 144.2 KiB | Viewed 2773 times ]
Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Tue Jul 10, 2018 7:55 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 917
Location: Europe
So which commands are you running?

.srdata is the volume where all performance metrics are stored, so if you are looking at historical data it is expected that there is activity on .srdata and as always you want the response as quickly as possible so it will take what performance it can.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Tue Jul 10, 2018 8:09 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
So which commands are you running?

.srdata is the volume where all performance metrics are stored, so if you are looking at historical data it is expected that there is activity on .srdata and as always you want the response as quickly as possible so it will take what performance it can.


Thanks for the reply..

That particular command was statvv so I can see which VV are generating the most I/O.. When I was having the slowness issue, which lasted for hours, the total I/O was averaging between 3K-5K..


Attachments:
2.jpg
2.jpg [ 198.09 KiB | Viewed 2752 times ]
1.jpg
1.jpg [ 116.71 KiB | Viewed 2752 times ]
Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Tue Jul 10, 2018 9:26 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
In CLI, «statpd -rw», «statvv -rw -ni», «statvlun -ni -rw» and «statvlun -hostsum -ni -rw» should give you some pointers. If statvlun is high and statvv is low, your issue is 99% host or fabric issue. With and without hostsum can help you understand if it is volume or host related. If statvv is high, the latency is due to something internal on the 3PAR, if statpd latency is high, it is most likely a backend issue (not enough disks/SSD). Always remember that read iops (vlun and vv) is mostly read from disk (around 5-10msec) while write should hit cache (1-ish msec).


So are you saying when running these commands to focus and compare the Svt MS columns?

Thanks


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Mon May 13, 2019 12:25 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
In CLI, «statpd -rw», «statvv -rw -ni», «statvlun -ni -rw» and «statvlun -hostsum -ni -rw» should give you some pointers. If statvlun is high and statvv is low, your issue is 99% host or fabric issue. With and without hostsum can help you understand if it is volume or host related. If statvv is high, the latency is due to something internal on the 3PAR, if statpd latency is high, it is most likely a backend issue (not enough disks/SSD). Always remember that read iops (vlun and vv) is mostly read from disk (around 5-10msec) while write should hit cache (1-ish msec).


Hello Mamma.. Back at it with my problem SAN.. So we added (8) 480GB SSD.. We manually put a few things in to the newly added SSD.. Our policy is to not enable AO on any of our SANS (Please don't ask me why.. it's frustrating to even talk about lol)..

Still having issues every now and again.. One thing I noticed is that the SVT time when I run: statvlun -hostsum -ni -rw .. ALmost all my hosts are over 20ms..

Attachment:
statvlun -hostsum -ni -rw-2.jpg
statvlun -hostsum -ni -rw-2.jpg [ 401.39 KiB | Viewed 1393 times ]


When I run statvv -sortcol 2,dec -ni all the svt times are under 1ms, for the most part..


I believe you mentioned that if statvlun is high and statvv is low, that it's a host or a fabric issue..

Can you recommend what I need to look at, at this point?

Thank you.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Mon May 13, 2019 2:02 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 917
Location: Europe
What does statport -host -rw say?

Can you provide some info about fabric?

And to refresh my memory... 3PAR OS, host OS and did you fix path policy?

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Mon May 13, 2019 2:14 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
What does statport -host -rw say?


Attachment:
statport -host -rw.jpg
statport -host -rw.jpg [ 128.54 KiB | Viewed 1390 times ]


MammaGutt wrote:
Can you provide some info about fabric?


Running iSCSI (puke).. 3Par is connected to a Cisco 4510.. Ports 1 & 2 are used on each Supervisor card, One to the SAN, One to the C7K, Both are Dual homed.. The switch has no other 10Gb capability and is physically maxed out of slots..


Te5/1 c7000 VC Trunks connected trunk full 10G Unknown SFP+-*
Te5/2 3Par-SAN connected 255 full 10G 10GBase-SR
Te5/3 3Par inactive 255 full auto 10GBase-SR < Not Usable
Te5/4 3Par inactive 255 full auto 10GBase-SR < Not Usable
Te6/1 c7000 VC Trunks connected trunk full 10G Unknown SFP+-*
Te6/2 3Par-SAN connected 255 full 10G 10GBase-SR
Te6/3 inactive 1 full auto No XCVR < Not Usable
Te6/4 3Par inactive 255 full auto No XCVR < Not Usable


Only 10GB Ports 1& 2 on the supervisor cards are enabled.. Ports 3& 4 are not,

In redundant mode, the Catalyst 4507R or 4510R chassis are deployed with dual Sup7-E supervisor modules in a redundant configuration. Port group 2 becomes automatically inactive on both supervisor modules when the Catalyst 4500E system detects redundant modules installed in the chassis.


MammaGutt wrote:
And to refresh my memory... 3PAR OS, host OS and did you fix path policy?


3PAR OS: 3.2.2.709 (MU6)+P99,P107,P119,P131,P135,P137,P138
Host OS: ESXi 6.5.0, 8935087, ESXi, 6.5.0, 13004031, ESXi, 6.5.0, 10719125 (not sure why these aren't all the same version. Need to follow up on that..

Pathing policy is Round Robin.

Thanks


Top
 Profile  
Reply with quote  
 Post subject: Re: 3Par Rookie - Need assistance in troubleshooting please.
PostPosted: Wed May 15, 2019 5:05 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 917
Location: Europe
Congratulations on the iSCSI and Vmware infrastructure :)

Google will give you a couple of hints that you're not alone :)

My suggestion... try everything ... Disable delayed ack, use software iSCSI, use hardware iSCSI and see where you get the best performance. And don't be surpriced if some servers perform best with software and others with hardware if you have different adapters (even from the same vendor) ... and hardware iSCSI can sometimes run into some "interesting" performance scenarios when simply upgrading the driver.

Also be aware that many network switches are highly oversubscribed and with very small port buffers. I've never seen the C4500 series on any list of recommended iSCSI switches :lol: but with the limited bandwidth you're using it shouldn't matter a lot.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 19 posts ]  Go to page Previous  1, 2


Who is online

Users browsing this forum: No registered users and 26 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt