HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: may 8gbit FC be the limit?
PostPosted: Thu Oct 29, 2020 5:35 pm 

Joined: Mon Dec 22, 2014 5:53 am
Posts: 5
Hi colleagues,

I have high saturation on 8200 (8x3,84 SSDs R55+1) array during nights when high data transfers happen on one VV. we limited qos to 500MB\sec for this VV and situation more or less good, at least not affecting another VVs
we have 8gbit FC ports
may be 8Gbit FC be the limit?
statport shows 99% average busy at the time of high saturation
--------IO/s-------- ---------KBytes/s--------- ----Svct ms----- ---IOSz KBytes---
Time Secs Rd Wr Tot Rd Wr Tot Rd Wr Tot Rd Wr Tot QLen AvgBusy%
2020-10-29 02:00:00 MSK 1603926000 1064.6 2508.4 3573.0 60613.8 542987.6 603601.4 6.63 41.39 31.03 56.9 216.5 168.9 63 99.6
2020-10-29 02:05:00 MSK 1603926300 2400.7 4315.5 6716.3 104573.8 630640.3 735214.1 2.48 16.66 11.59 43.6 146.1 109.5 64 99.0
2020-10-29 02:10:00 MSK 1603926600 1022.7 3272.1 4294.8 94691.7 552398.6 647090.3 3.85 19.68 15.91 92.6 168.8 150.7 25 98.9
2020-10-29 02:15:00 MSK 1603926900 1014.5 3211.6 4226.1 138686.4 548928.6 687615.0 4.06 12.17 10.22 136.7 170.9 162.7 62 97.8
2020-10-29 02:20:00 MSK 1603927200 861.8 3132.1 3993.9 172123.6 547136.5 719260.1 5.94 21.16 17.88 199.7 174.7 180.1 83 99.5
2020-10-29 02:25:00 MSK 1603927500 869.1 3043.7 3912.7 155233.1 551130.4 706363.5 6.64 31.39 25.89 178.6 181.1 180.5 130 99.7
2020-10-29 02:30:00 MSK 1603927800 532.4 3181.5 3713.9 171513.4 596924.1 768437.5 9.33 39.57 35.23 322.1 187.6 206.9 37 99.7


Top
 Profile  
Reply with quote  
 Post subject: Re: may 8gbit FC be the limit?
PostPosted: Fri Oct 30, 2020 9:05 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
8Gbit FC should be able to do 720MB/s per port.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: may 8gbit FC be the limit?
PostPosted: Fri Oct 30, 2020 11:15 am 

Joined: Mon Dec 22, 2014 5:53 am
Posts: 5
just wondering why AvgBusy is about 99% and the saturation at that moments quite high. disk AvgBusy near 60-70% at that moments.

Why host ports... we checked this bandwidth is load balanced across all 4 connected ports, load definitley lower than 8 Gbit\s )


Top
 Profile  
Reply with quote  
 Post subject: Re: may 8gbit FC be the limit?
PostPosted: Fri Oct 30, 2020 12:02 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
I'm just taking a really wild guess here.

QoS is limiting a VV that might be causing queue on the port which fools the AvgBusy% parameter. I would collect a performance analysis when it happens to review statport, statvv and statvlun to see where you have the queue and where it is.

Also, your output is very hard to read, but if I see 500MB/s write and you're DeCo volumes on a 8200, that might be a factor too.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: may 8gbit FC be the limit?
PostPosted: Mon Nov 02, 2020 8:56 am 

Joined: Mon Dec 22, 2014 5:53 am
Posts: 5
Yes, it's dedup and compression enabled, and 500MB/s it's the limit that we found experimentally, at that limit all other volumes not affected with big bulk writes on that VV.

Cpu load not higher than 50%,might that be due dedup slowness?


Top
 Profile  
Reply with quote  
 Post subject: Re: may 8gbit FC be the limit?
PostPosted: Mon Nov 02, 2020 9:10 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
kolin wrote:
Yes, it's dedup and compression enabled, and 500MB/s it's the limit that we found experimentally, at that limit all other volumes not affected with big bulk writes on that VV.

Cpu load not higher than 50%,might that be due dedup slowness?


Are you looking at total CPU or per core? Not everything multithreads perfectly.

Also, that 500MB/s limit... Did you account for the following bits in the works that led to that number:
1. One frontend read IO equals one backend IO. (frontend is host ports, backend is SSDs)
2. One frontend write IO equals 4 backend IOs with RAID5 and ~6 1/2 backend IOs with RAID6
3. With dedupe, one frontend write IO that dedupes equals one backend IO (read to do block compare)
4. With dedupe, one frontend write IO that doesn't dedupe = one frontend write IO without dedupe.
5. 500MB/s with 4kB IO size = 125.000 IOPS, 500MB/s with 64kB IO size = ~8.000 IOPS

Considering that you don't measure the read/write ratio, you can't define dedupe hit ratio and you don't know if your limit is bandwidth or IOPS (or a combination) .... setting a max bandwidth sounds pretty risky.... I would probably rather use latency goals or similar to priortize everything else with the limited information provided here.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 


Who is online

Users browsing this forum: Google [Bot] and 79 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt