HPE Storage Users Group
https://3parug.net/

4-node 7450c - POR STATE Leds and internal error register
https://3parug.net/viewtopic.php?f=18&t=3560
Page 1 of 1

Author:  GrantJoyce [ Fri Apr 02, 2021 8:49 pm ]
Post subject:  4-node 7450c - POR STATE Leds and internal error register

Hi 3parugers,

Please, if I could borrow your 3par combined wisdom to help me bring life back to a 3par 4-Node 7450c system I purchased 2nd hand. One node (node-3) boots ok. I can connect to the console at least and I can login as root and console.

The other 3 nodes (nodes 0,1,2) do nothing, except blink the fault and hotplug leds. No console. Never does any external led flash green.

Internally, on the node mainboard, beneath where an expansion card sits, there are 5 additional diagnostic leds, labeled as POR STATE. I can see these by removing the expansion card and connecting the node without its cover. Only leds 0 and 1 light green on the dead nodes (0,1,2). On node 3, leds 0,1,3,4 light green.

My thinking is, that for 3/4 nodes to apparently all fail simultaneously, all having the same symptom, but still functioning diagnostics, all saying the same. Either something catastrophic has occurred, damaging 3/4 nodes at the same time in the same way, which seems pretty unlikely. Or, some event has caused the entire cluster to panic, placing 3/4 nodes into some sort of defensive state in order to avoid potential, actual failure.

Could you offer any insight into what this tells me, if this may be the case, and how I might recover these 3 nodes? I am hoping there be a procedure to clear or reset the nodes internal error register somehow.

Many thanks in advance.

Author:  MammaGutt [ Sat Apr 03, 2021 2:36 am ]
Post subject:  Re: 4-node 7450c - POR STATE Leds and internal error registe

My first thought is when did these fail? Have they ever worked for you?

Just to verify you are getting no console output at all on the three nodes not working?

Author:  GrantJoyce [ Sat Apr 03, 2021 3:36 am ]
Post subject:  Re: 4-node 7450c - POR STATE Leds and internal error registe

I bought "As-Is" a 3par rack containing a 4-Node 7450c ( E7X62A ), having 4x E7X87-63001 Controller modules, 8 x HP M6710 ( QR490A ), and a DL320e SP. No disks were supplied. I am new to 3par. My understanding has been gathered entirely over the past week, trying to get this kit operational.

I've never seen this kit operating, so I may have purchased 3 dead nodes, and 1 maybe ok / maybe not node. I have cabled up just the 4 Nodes. Power, network, io, and powered the lot on. And what I describe here, is how this I received this kit.

Node 3, power on, almost immediately flashes some green. Leds for network, fibre (onboard + 4x 8gb expansion), all flash some green and look healthy. IO connector leds appear healthy, although saying no link. With a console cable connected, I get output and can logon.

Nodes 0,1,2, Power on, never do leds flash, except for Hotplug and Fault, which flash together at the same time. The controller will power on, flash those two leds a few times, then go dark for a few seconds, then come back on blinking forever. With a console cable connected, I get zip. Messing about with the console, unplugging, plugging back in, etc. sometimes I get a scrambled character. One time, I got a quarter page of scrambled characters. Progress I thought, more is better than less right?

Internally on these nodes, there is 5x led for POR STATE (Power-On-Reset), which show leds 0,1 green. Node 3, these same leds show 0,1,3,4 green.

I have noticed that on the enclosure for Nodes 0 and 1, both battery leds show failed. Enclosure for Nodes 2 and 3, appear ok, however, initially, the battery led indicated warning / sw fixable fault, but that seems to have cleared.

I have swapped the ssd-boot disk between nodes. Node 3 boots ok using ssd-boot from other nodes, but not vice-versa. I have connected Node 3's ssd-boot disk to my linux workstation via SATA cable, and noticed the log partition was 100%. Errors were throughout recent history regarding unavailable space to write logs, so I deleted a 36Gb system log archive.

I have tried swapping all parts from Node 3 with another Node. Node 3 always boots ok, other nodes do not. Always same issue. blinking hotplug and fault.

I have tried removing / swapping the button battery. Currently I have one node with button battery removed, I am hoping that eventually whatever it was keeping charged will reset, and maybe clear whatever is stopping it from posting.

My next step I suppose is to examine the logs on the other nodes.

Author:  MammaGutt [ Sat Apr 03, 2021 6:16 am ]
Post subject:  Re: 4-node 7450c - POR STATE Leds and internal error registe

Sounds like dead nodes to me.

Author:  GrantJoyce [ Sat Apr 03, 2021 7:24 am ]
Post subject:  Re: 4-node 7450c - POR STATE Leds and internal error registe

I am thinking much the same.

I am now reflecting upon how I came to acquire this machine. I bought it as a complete system from auction. No doubt, the previous owner went out of business when their super redundant, mission critical, best in class, 240 All-Flash disk array went belly up.

Can't blame the IT decision makers really. Who would ever expect 3/4 nodes of an enterprise storage system failing simultaneously, for no apparent reason, when they are specifically designed entirely around that NOT happening.

Perhaps, if I can get another 12 nodes, and 24 disks enclosures, at a 3 in 4 failure rate, I might put a running system together.

And at that fail rate, we should expect to see more of these come available, as the liquidators move in.

Author:  MammaGutt [ Sat Apr 03, 2021 9:38 am ]
Post subject:  Re: 4-node 7450c - POR STATE Leds and internal error registe

My experience is that old storage systems tend to have latent hardware issues displaying themselves when power is removed and/or systems are transported.

It wouldn't be surpriced if the system was working when it was shutdown. For you to buy a system without drives gives you and/or the broker a chance to verify if the system is really operational until you get a hold of drives.

Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/