HPE Storage Users Group

Hi,

I'm new to this forum and new to 3Par so hopefully I will use the correct terminology.

We have a 7440c with SSD and FC/SAS drives only. In setting up the CPG's I see that the best practice for FC/SAS and for the SSD is RAID 5 with a default set size of 3+1. Why wouldnt the best practice be RAID 6? I understand you loose another drive to parity but you get the added protection of another drive failure on the odd chance that two drives in the same RAID5 set fail before it can be rebuilt. I also understand there is a write penalty with the extra parity. I also understand the rebuild will use all the drives to rebuild as it reserves space on each drive making the rebuild faster. I also understand it only rebuilds the used space and does not rebuild the white space.

My thinking is this, lets say I take the recommended RAID 5 set of 3+1 for my FC/SAS CPG. If two drives in the same RAID 5 set fail before it rebuilds I have lost whatever chucklets where on that RAID set. Now comes the hard part, since data on the 3par is spread out as chucklets you really wont know what to restore. Its not like I can go to my backups and restore the chucklets as I dont know what was there. The only thing I can think of is I would go to the latest snapshot prior to the failure but you would lose anything between the snapshot and double disk failure.

Am I missing something here or are others configuring your CPG's for FC/SAS and SSD's as RAID 6?

thanks

With an ~1.5% AFR and the way chunklets are laid out (especially in cage availability mode) and the very fast rebuild with every disk to every disk system that the chunklets plus ASIC enable the chances of a double fault resulting in dataloss are so small that I doubt it's ever happened on any 3Par. In fact according to a calculator I found the average time to data loss due to multi-disk failures given a 1200GB disk, a 300MB/s restore speed, and a 1.5% AFR is 6,488,889 years for RAID5 3+1 with online hostspare (which the 3Par has through spare chunklets). This of course assumes pure independence between disk failure events, something not fully realistic in the real world, but the order of magnitude is so great as to underscore the slim possibility of it happening outside some freak occurrence.

Double Drive failures are rare, but do happen, statistically if you have an array with many drives then raid 3+1 means you would have to have 2 drives in the same 4 drive set fail, even more statistically unlikely. Also the way chunklets are scattered across disks you data is not always not he same 4 disks. Sure you can always use Raid 6, but the write penalty is severe. HP really pushes Raid 6 for large NL drives because of the rebuild times. We have numerous 2TB drives and rebuild times can be 24+ hours when the system is busy yet we have been Raid 5 for 3+ years. We do plan on taking the NL to Raid 6 we just have not had time.

hdtvguy wrote:Double Drive failures are rare, but do happen, statistically if you have an array with many drives then raid 3+1 means you would have to have 2 drives in the same 4 drive set fail, even more statistically unlikely. Also the way chunklets are scattered across disks you data is not always not he same 4 disks. Sure you can always use Raid 6, but the write penalty is severe. HP really pushes Raid 6 for large NL drives because of the rebuild times. We have numerous 2TB drives and rebuild times can be 24+ hours when the system is busy yet we have been Raid 5 for 3+ years. We do plan on taking the NL to Raid 6 we just have not had time.

I know old post, but as information isnt correct in this i will add the correct info.

The 3par raid paritiy isnt per set, its per tier / CPG. This is due to the distributed parity. So any 2 disks need to fail to cause data loss in RAID5. it is a raid 5 with chunklets, these are spread over every disks, so if 2 complete drives fail, you will lose data. Its only if specific sectors fail on the disks that you are less likely to lose data as it then covers the chunklets in the sets.

This I realized the hard way. On 2 3pars, with sync replication. Both had 2 disks fail at the same time. Data lost as it was in RAID 5, but luckily they were affecting different luns.

david wrote:

hdtvguy wrote:Double Drive failures are rare, but do happen, statistically if you have an array with many drives then raid 3+1 means you would have to have 2 drives in the same 4 drive set fail, even more statistically unlikely. Also the way chunklets are scattered across disks you data is not always not he same 4 disks. Sure you can always use Raid 6, but the write penalty is severe. HP really pushes Raid 6 for large NL drives because of the rebuild times. We have numerous 2TB drives and rebuild times can be 24+ hours when the system is busy yet we have been Raid 5 for 3+ years. We do plan on taking the NL to Raid 6 we just have not had time.

I know old post, but as information isnt correct in this i will add the correct info.

The 3par raid paritiy isnt per set, its per tier / CPG. This is due to the distributed parity. So any 2 disks need to fail to cause data loss in RAID5. it is a raid 5 with chunklets, these are spread over every disks, so if 2 complete drives fail, you will lose data. Its only if specific sectors fail on the disks that you are less likely to lose data as it then covers the chunklets in the sets.

This I realized the hard way. On 2 3pars, with sync replication. Both had 2 disks fail at the same time. Data lost as it was in RAID 5, but luckily they were affecting different luns.

You are also not 100% correct

If you have multiple node pairs, each node pair is a failure domain. If you have HA cage, all drives in a cage could be considered "one drive".

But RAID5 have never been more secure than to withstand one failure. It can survive more under different scenarios and configurations, but worst case is always data loss on failure number 2.

The 3PAR RAID parity is per RAID set, but each RAID set consists of chunklets, not disks. So given enough data, all drives in the same failure domain will share some RAID set with every other drive in that failure domain.

MammaGutt wrote:
You are also not 100% correct

If you have multiple node pairs, each node pair is a failure domain. If you have HA cage, all drives in a cage could be considered "one drive".

But RAID5 have never been more secure than to withstand one failure. It can survive more under different scenarios and configurations, but worst case is always data loss on failure number 2.

The 3PAR RAID parity is per RAID set, but each RAID set consists of chunklets, not disks. So given enough data, all drives in the same failure domain will share some RAID set with every other drive in that failure domain.

So are you saying that with more node pairs the chunklet layout is different (I only have dual node arrays)? If not, then given that you are using any reasonable amount of data you will have overlapping chunklets within raid sets on any 2 drives, unless you have a significate number of very small drives, or you specify availability at the JBOD/shelf, then it would be any 2 disks in any 2 JBOD/shelf.
Back when this happened, it was also stated by HPE support that in certain circumstances RAID6 can see data loss with a 2 disk failure on the 3par.

This means that the availability of data is reduced due to the significantly higher chance of data loss in the event of 2 disks failing, unlike with standard RAID5 which with a large number of drives and a small RAID set would have a lower probability of dual drive failure within the same set. So the increase in performance that the chunklet method provides comes at the cost of increased chance of data loss. Which no doubt is hoped that the increased speed of rebuild and reduced load on all disks is hoped will not cause a second drive to fail before it is complete.

This will also be the same for mirror any 2 failed disks will likely cause data loss. Again unlike normal RAID 1, availability at the cost of performance, spreading the chunklets over all the drives.

Each pair of nodes would have a separate set of trays/disks connected to it. So if the system lost 2 disks they might be in opposite node pairs, so the the equivalent of only a single disk failure on a two node system.

Of course you still might be unlucky and the 2 disks fail behind a single node pair.

Mix 4 nodes with cage parity and on the older V class arrays you could loose 80 drives at once (one cage per node pair) and still be fine with RAID5. Although loose one more disk anywhere (probably due to the stress of the parity being rebuilt ) and some VV would loose data. However not all VVs with loose data as unless the VV is very very large it won't live on 'every' disk, just parts of it will live on parts of disks on a selection on disks from the array.

ailean wrote:Each pair of nodes would have a separate set of trays/disks connected to it. So if the system lost 2 disks they might be in opposite node pairs, so the the equivalent of only a single disk failure on a two node system.

Of course you still might be unlucky and the 2 disks fail behind a single node pair.

Mix 4 nodes with cage parity and on the older V class arrays you could loose 80 drives at once (one cage per node pair) and still be fine with RAID5. Although loose one more disk anywhere (probably due to the stress of the parity being rebuilt ) and some VV would loose data. However not all VVs with loose data as unless the VV is very very large it won't live on 'every' disk, just parts of it will live on parts of disks on a selection on disks from the array.

Taking it to the extreme. 8node v-class with 480 drives (3cages/120 drives per node pair) in R5 (2+1). You could lose 160 (1/3) drives and be okay or you could lose 2 drives in different cages behind the same node and lose data.

Physical disks are divided into chunklets. Each chunklet occupies physically contiguous space on a FC or NL disk. On all current HPE 3PAR StoreServs all chunklets are 1 GB. Chunklets are automatically created by the HPE 3PAR OS, and they are used to create logical disks. A chunklet is assigned to only one logical disk https://jealouscomputers.com/.