adding expansion drive enclosures (7400)

[Rugbyman]

Hi

We are looking at adding 4 additional drive enclosures to our 7400 (4 node). We currently have 10 additional enclosures.
I'm just looking for some guidance on the best way to do this:
We will add 2 at the top of the enclosure, and 2 at the bottom.

Looking at the StoreServ installation guide, it looks pretty straightforward, except for the cabling:
http://h20628.www2.hp.com/km-ext/kmcsdi ... 6625-9.pdf

To add expansion drive enclosure:
1. Install the expansion drive enclosure. See “Installing Enclosures ” (page 25).
a. Install the disk drives. See “Installing a Disk Drive” (page 28).
b. Cable the enclosures to each other by using SAS cables. See “Cabling the Storage
System” (page 38).
NOTE: For the drive enclosures, verify that the activity LED is functional (all four LEDs will
be lit solid green), and the LED at the front of the enclosure has a number.
2. Install the 580 W PCMs into the drive enclosure.
3. Install the power cables to the PCMs and press the power switch to ON.
4. After you have completed the physical installation of the enclosures and disk drives, cable the
drive enclosure to the controller nodes.
5. Verify the upgrade is successful.

I understand 1a, my question is around the cabling.
As Node 1 is currently connected to the bottom 2 shelves, would you remove these 2 cables from the bottom shelves, add the SAS cables between the old and new shelves, power the new shelves on, and then reconnect the controller cables to the new bottom 2 shelves?

Then do the same with the top shelves?

[Davidkn]

Great topic, and I'm just going through this myself, I'm a consultant working for a reseller and actually deliver the hp services, and even we haven't got access to this information, it's not great.

I've done upgrades before but not had to remove the sas cables in node 1, because as you say they currently go to the bottom shelves, but when adding 2 new shelves to the bottom, the cables then need to go to those new shelves.

For the FC shelves on the 10,000 series, there's a command to stop io on the controller to those ports, using servicecage startfc, but I can't find the same info on the sas shelves.

I have a call open with hp at the moment to get this info, so as soon as I get it I will share.

My take on it would be to cables the node 0 cables and the intranet shelf cables, and then finally move the node 1 cables, but I can also see how moving these cables first could also make sense.

Another thing to mention is that once you've added the shelves, they might need a firmware update, to start this off you run the upgradecage -a command.

You probably want to put it maintenance mode too while disconnecting the cables so that it doesn't go mental with alerts to hp.

Are you not working with hp for the install, did you not purchase the installation lines with shelves? We have another shelf upgrade for another client and hp have told us that before we can do the upgrade they need to update the sp and add some patches to the inserv too.

So it might be worth checking with hp first to make sure there's no software/firmware upgrades that need doing first.

David.

[Rugbyman]

Hi David

Addittional shelves have not been ordered as yet, hopefully in the next couple of weeks.

The controller cables are alerady in place as it is a 7400 (4 node)
node 0 is already cables to the shelves directly below it, so i dont think i need to move these.
Thinking you just need to move node 1 and node 3 cables??

[BryanW]

Just curious, are you in a spot where it is difficult or cost prohibitive to have an HP implementation engineer install the enclosures? The installation service is not particularly expensive and the SAS cabling at the nodes is kind of a spaghetti bowl. The HP DCIS engineers have a giant document with all of the potential cabling configurations and a script they reference throughout the process.

That said if you need I could take a pic of what the loop cabling of a 16 enclosure configuration looks like if you need it.

FYI, if the enclosures are on a different firmware revision than the array it will upgrade/downgrade the new ones as part of the upgrade and it will take 20-30 minutes per enclosure.

[Rugbyman]

Hi Bryan

Kit not ordered yet, I'm just interested how it's all done.
The cable layout is not a problem as all the shelves are SFF, one drive type and one size.

[Davidkn]

I'm just waiting for my plan from hp, I got one through last night for another job we are doing but that was just adding the first shelf to a controller shelf, so no cables to remove and change.

When I get mine through I will let you know what it says, the doc is pretty good, it goes through everything like a script as mentioned above, but I think you can only get this if you purchase hp services to do it, which can either be implemented by HP or the reseller (like in our case) if they are a service one partner.

P.s. Let me know if you want a competitive quote to compare

[Davidkn]

So I finally did my upgrade today.

It was going to be delayed, as they get you to update the sp and 3par OS to make sure the new disks are supported.

I argued the case that we didn't have time and so in the end we just added a patch to support the drives after I found out the drive model being added (hp couldn't tell me, I had to do a physical check, they tell you to update to the latest to cover all bases basically. )

I got the upgrade plan, and this ran through everything step by step, and even took into account the fact that the drive shelf numbers were out of order, so it's tailored for your environment.

It didn't follow what is posted in the op, you run a checkhealth, rack the new cages, add in the disks, then apply power and make sure all the drives light up properly.

Then do a locate cage, and do the cabling according to the plan, using showcage throughout to make sure the shelves are being seen properly.

Then once all disks are being shown as normal and on two ports, you run an admithw which does any firmware upgrades (or downgrades in my case on the new cages that were added).

At the beginning you also put the nodes into maintenance mode to stop alerts (although as soon as I enabled it again the customer received a load of emails so it doesn't seem to do much)

Not sure I can post the actual plan as it's customised, I might try and make it generic or something. It would only really help on the cabling if you had a 7200 with 3 additional shelves and were adding 2 shelves and the original shelf order was 0, 2 and then 1.

I'd recommend contacting hp though to see if they'll do a plan for you.

[Davidkn]

Oh and I had something weird happen, the cpg options changed after the upgrade, very strange.

Before the upgrade I had a cpg that was for 900gb 10k disks, mag availability, raid 5 with 5+1

After the upgrade it changed to cage availability and 3+1, which wouldn't have been possible before the upgrade as I only had 3 shelves.

As I was adding 300gb 15k drives, I didn't want these to be used by the existing cpg, and as it had been set up with default drive speed it meant that it could contain the 10 and 15k drives. So I tried to edit it to be 10k only, and when I did the next button was greyed out and it said I dint have enough space. And the reason was that the policy was this 3+1, and I did have 4 shelves, but only 3 with 10k drives.

So I had to change it to mag availability and 5+1 ( the same as what it was before the upgrade) and then it let me specify 10k drives.

Very strange, the hp guy had never seen it do this either.

Worth checking your cpgs after an upgrade though.

[Richard Siemers]

Thanks for the heads up. I just upgraded from 2.3.1 --> 3.1.1 --> 3.1.3 MU5 on a T800, However no adding of cages. Your post worried me and I had to go verify my CPGs were still set the way we want them and all is good, no hijacking here. There were the default cpgs created though... also on your note about maintenance mode... I had the same issue... we put the T800 in maintenance mode to suspend alerting however after the upgrades were completed, we removed maintenance mode and BAM an avalanche of pages blew up my phone. Disturbing how different the ring tone sounds when its played 20 times simultaneously...

[Davidkn]

That's quite an upgrade!

Presume it all went well then.

What this cpg issue highlighted to me was to note down the settings of the cpgs before performing the upgrade in future, as if I hadn't had hp on the phone telling me that the data they had before the upgrade was different, I probably wouldn't have noticed. As I wouldn't have even thought anything like that would happen, especially with just a disk upgrade.