HP 3PAR 7200 Startup error

[Balkoiner]

Hello everybody. I have a problem. Could you help me?
I have a 3PAR 7200. I try to run it after full rebuilding. I connect to it through SSH and console cable.
The result of command showsysmgr:
System is attempting recovery from a previous power failure.
Please use the showvv, showld, showpd CLI commands to
check for not started VVs, LDs, and PDs. Unstarted VVs
can cause recovery to stall.
The result of command showvv:
1 .srdata full base --- 1 RW not_started 0 0 61440 61440
0 admin full base --- 0 RW lds_not_started 0 0 10240 10240
Other VV is normal state. All pd is degraded.
Does anyone some ideas how to fix it?

[MammaGutt]

If all PDs are degraded, I'm taking a stab that you are missing a node in your system.

[Balkoiner]

All of two nodes in normal state
Node --Name--- -State- Master InCluster -Service_LED ---LED--- Mem(MB) Mem(MB) Available(%)
0 xxxxxxx-0 OK Yes Yes Off GreenBlnk 8192 4096 100
1 xxxxxxx-1 OK No Yes Off GreenBlnk 8192 4096 100

Result of checkhealth:

Component -----------------Summary Description------------------ Qty
Alert New alerts 9
Cabling Wrong I/O module or port 12
Cage Degraded or failed cage power supplies 4
Cage Degraded or failed cage AC power 4
cert Certificates that have expired 1
File sr_mnt not mounted 1
File Admin Volume is not mounted 1
LD Failed or degraded LDs 1
LD LDs with reduced availability 3
License Licenses which have expired 1
Network Too few working admin network connections 1
Component Could not check node 1
Component Could not check pd 1
pdch LDs with connection path different than ownership 198
pdch LDs with chunklets on a remote disk 198
Port Unavailable host ports 4
QoS Unable to check QoS 1
Component Could not check task 1
vlun Hosts not connected to a port 34
VV CPGs unable to grow SD space 1
VV Degraded or failed virtual volumes 2
VV Volumes which have reached user space allocation limit 1
--------------------------------------------------------------------
22 total 480
3PAR was totally demount. All drives and cage was unmount, without any remarks. And now it mounted in another sequence. I'm sure that all positions of nodes, cages and drives messed up.

UPDATE: After 24H system restored. But all pd still in state degraded. Maybe i must run servicemag for all pd's or maybe i should wait for some time?

[MammaGutt]

This will not fix itself and servicemag will not help.

When mixing up positions etc on external drive cages, everything goes to shit

All 3PAR come with a few sheets of paper when labels for all SAS cables. They should have been labeled during install and re-labeled during any upgrade. There are detailed schematics available at HPE for basically any scenario.

So if all that is missed and in addition no one cares to label any cables before de-install... you are asking for trouble and that's what you got

[Balkoiner]

I understand that it smells like large of shit, but it not depended from me. And i can't rewind back. Can i do something to fix it. Please don't say that i have only one way - totally reset system.

[MammaGutt]

I would reach out to HPE Support or some consultancy with 3PAR skills. One would have to go thru the actual issues you're seeing here and do a clean-up.

My guess is that the mis-cabled cages have been admitted to the system and ghost IDs needs to be dismissed. You also probably has some LDs that needs to be re-tuned. Lot's of chunklets needs to be moved (that might fix itself) ++++ But that's just me taking a guess based on what information is here.

[Balkoiner]

In management console id's begin from 0. I think that means it's totally new id's without any old record's, but they all degraded. Anyway i open case in HPE support. Thanks for help.

[Balkoiner]

I have new information. Probably it can help with my problem.
I create VSP and it's give me right positions of cage's i move it. After reboot system all disk in cage's are normal, besides disks in cage0. All disk in cage 0 shown next error:
PD disk:3 Degraded States: Invalid_connection

How do you think, can i fix this or problem with hardware?
Probably i should remove disks and re-add them with removing spares?
Like that:

setpd ldalloc off <DiskID>
movepdtospare -f -vacate -nowait <DiskID>
showpdch -mov (too see status/what is left, if it doesn't complete, you can just just tunesys cpg <NL CPG>)
removespare PDID:a
dismisspd <DiskID>

Remove PD
Reinstall PD
admithw

[Balkoiner]

Thanks for help. It fixed by commands in previous post. I spent above 60 hours, but it helped me