Adaptive Optimisation - CPG Config with varying drive sizes

[gingerprince]

Hi

We have a 4 controller V400 which presently has 240 x 300GBFC15 spindles (raid5-3.1) and 48 x 2TBNL72 spindles (raid6-6.2). Writes go into FC and AO manages migration down to NL.

We are doubling the capacity by adding another 48 NL spindles (which will go on the NL CPG) and 144 x 600GBFC15 spindles. We have 2 conflicting options in terms of onwards configuration. I have asked our HP team for advice, but looking to the community for other thoughts. We can either: -

* Add the 600GB spindles into the existing FC CPG. Once we tunesys to balance, the 600GB drives will have more capacity than the 300's. As new data is written it'll have 384 spindles to go at, but once the 300GB drives are full any new writes will be limited to 144 spindles. Performance in this tier will therefore be unpredictable.

* Create a "middle" tier, such that the existing 240 x 300FC drives are tier-0, the 144 x 600FC drives are tier-1 and the 96 x 2TBNL drives are tier 2, with writes going into tier-0 by default.

In the second example, what I don't know is will AO cope? I don't think there is enough information out there regarding the statistical calculations that are used to determine whether data is moved up/down tiers, and whether the fact there are essentially 2 tiers that are both 15K FC will confuse matters. I can see it would work with 10K drives in the middle, but 15?

What would you do? Does anyone do either of the above? Does anyone know how AO would cope?

Thanks

[hdtvguy]

I would split them into a separate group. My concern would be if 600 and 300 GB drives are in same CPG as the blocks are evenly written the 300GB drives will fill up and space will remain on the 600 GB drives. if you have CAGE HA you may wind up in scenario where data may not be Cage HA. There are pros an cons to each scenario, you need to determine what is important to you. We find that unless you apps are very hot AO will move data down even when set to Performance.

[hdtvguy]

This is what HP Support provided me on how AO works:

Setting mode as Performance will move as much data as possible to higher tiers but it does not mean that data will not be moved to lower tiers.
In your case, we see the AO is working as per design, even in idlest Performance and zero performance also data is not getting moved to lower tier it means that there are no idle data or zero performance data which can be moved to lower tiers.

I have also explained below how AO is designed to work for your better understanding. AO uses certain algorithms to decide the region moves between the tiers. All listed below.

The AO moves the regions between tiers based on certain algorithm. It uses below 3 algorithm to move data between tiers.

1. Space available in the tiers.

In this, If the space in a tier exceeds the tier size (or the CPG warning limit), then the algorithm will first try to move regions out of that tier into any other tier with available space in an attempt to lower the tier’s size below the limit. If no other region has space, then the algorithm logs a warning and does nothing. (Note that if the warning limit for any CPG is exceeded, the array will generate an alert.) If space is available in a faster tier, it chooses the busiest regions to move to that tier. Similarly, if space is available in a slower tier, it chooses the most idle regions to move to that tier. The average tier service times and average tier access rates are ignored when data is being moved because the size limits of a tier have been exceeded

2. Average tier service times.

Normally, HP 3PAR Adaptive Optimization tries to move busier regions in a slow tier into higher performance tiers. However, if a higher performance tier gets overloaded (too busy), performance for regions in that tier may actually be lower than regions in a “slower” tier. In order to prevent this, the algorithm does not move any regions from a slower to a faster tier if the faster tier’s average service time is not lower than the slower tier’s average service time by a certain factor (a parameter called svctFactor). There is an important exception to this rule because service times are only significant if there is sufficient IOPS load on the tier. If the IOPS load on the destination tier is below another value (a parameter called minDstIops), then we do not compare the destination tier’s average service time with the source tier’s average service time. Instead, we use an absolute threshold (a parameter called maxSvctms).

3. Average tier access densities.

When not limited, as described above, by lack of space in tiers or by high average tier service times, adaptive optimization computes the average tier access rate densities (a measure of how busy the regions in a tier are on average, calculated with units of IOPS per gigabyte per minute) and compares them with the access rate densities of individual regions in each tier. Then, it decides whether to move the region to a faster or slower tier.

We first consider the algorithm for selecting regions to move from a slower to a faster tier. For a region to be considered busy enough to move from a slower to a faster tier, its access rate density and accr(region) must satisfy these two conditions:
1. First, the region must be sufficiently busy compared to other regions in the source tier:
accr(region) > srcAvgFactorUp(Mode) * accr(srcTier) Where accr(srcTier) is the average access rate density of the source (slower) tier and srcAvgFactorUp(Mode) is a tuning parameter that depends on the mode configuration parameter. Note that by selecting different values of srcAvgFactorUp for performance, balanced or cost mode values HP 3PAR Adaptive Optimization can control how aggressive the algorithm is in moving regions up to faster tiers.
2. Second, the region must meet one of two conditions: It must be sufficiently busy compared with other regions in the destination tier, or it must be exceptionally busy compared with the source tier regions. This second condition is added to cover the case in which a very small number of extremely busy regions are moved to the fast tier, but then the average access rate density of the fast tier create too high a barrier for other busy regions to move to the fast tier:
accr(region) > minimum((dstAvgFactorUp(Mode) * accr(dstTier)), (dstAvgMaxUp(Mode) * accr(srcTier)))

The algorithm for moving idle regions down from faster to slower tiers is similar in spirit—but instead of checking for access rate densities greater than some value, the algorithm checks for access rate densities less than some value:
accr(region) < srcAvgFactorDown(Mode) * accr(srcTier) and
accr(region) < maximum((dstAvgFactorDown(Mode) * accr(dstTier)), (dstAvgMinDown(Mode) * accr(srcTier)))

HP makes a special case for regions that are completely idle (accr(region) = 0). These regions are moved directly to the lowest tier.

AO uses all the algorithms when performing the moves. Whichever algorithm suites best, it choses that mechanism to decide the movements. The Space algorithm will be considered only when there is a space crunch. The rest of the algorithms will be used every time the AO runs. In your case except the space algorithm, rest all algorithms will be used to decide the movements.

Note : AO Metrics are purely depends on your AO setup Environment and configuration. There will not be any fixed or defined values for metrics, its changes as per your AO setup.

These are internal to the AO software and can’t be set by user. The Inserv will decide all these parameters based on the sample collection. It varies depending upon the load on the inserv and the physical disks.

With Inform OS 3.1.1 or earlier, in a given schedule, AO can move only 1TB of data between tiers. With on Node AO you can move more than 1 TiB or 5% of total AO config capacity. However, without looking to the AO config we cannot decide hence the statement as the warnings may be set at CPG level and we all know that AO respects the warning or limit set at AO CPG.

[afidel]

With 3.1.2 MU2 or MU5 I have never had AO move more than 1TB per AO config per run. This was actually one of the factors extending the timetable for the rebalance after our recent upgrade.

[RitonLaBevue]

Don't get 300gb and 600gb in a single CPGs. You will run into serious troubles when 300gb drives are full. As you wrote it.
Tier 0 with 300gb R1, Tier 1 600gb R5 3+1 and Tier 2 2tb R6.

Did you really watch "sraomoves -btsecs -86400" daily ?
So far i never heard about 3.1.2 MU5 or even 3.1.2 MU4.

[afidel]

I was watching it daily as we were moving a large amount of data around to free enough blocks on our FC tier to move 32 drives from the original nodes to the two new nodes and the fact that it was only tiering down 1TB per night was impacting the scheduling.