HPE Storage Users Group

I had a failed node on my 3PAR 8400. I replaced the node and the replacement was successful yet a "healthcheck" reveals the following problem for cage 0 "Firmware is not current". Does this mean that the firmware on the replacement node does not match node 1. What should my next step be?


I was thinking about issuing the "showcage -d cage0" comand to check the firmware version of node 0 and node 1.

From there I would issue the "upgradecage cage0" command in hopes that it would catch node 0 up to the current firmware version of node 1.

Any thoughts?

admithw is a good command

How did you execute the replacement? Did you trigger a rescue node procedure?

The "shownode" command only showed 1 node as it was a complete node failure.

1. Before removing the defective node, press in on all cables on every node to ensure they are properly seated.
2. Verify the node to be replaced is halted (blue UID light will be solid)
a. If it’s not halted, issue a # shutdownnode halt <node_id> from another node’s CLI.
3. Replace the node. When pulling cables, make sure to either label them or tuck them off to the side from which you disconnected them. If the node disk is known to be good, swap the disk over to the new node. If unknown, do NOT move the disk. Move the HBA and SFP’s (if equipped) (don’t forget the onboard SFP’s).
4. Make sure all cables are full seated BEFORE seating the node all the way into the chassis, including your serial cable.
5. Watch the node boot from your serial console session. If you didn’t carry over the original node disk to the new node, you should see the following output:
Whack>set perm cnt_no_os_boot=0
Whack>set perm cnt_no_cluster=0
Whack>set perm cnt_no_shutdown=0
Whack>set perm cnt_os_panic=0
Whack>set perm cnt_same_fatal=0
Whack>set perm sys_serial_10=XXXXXXXXX
Whack>set perm w19=0x7E1F9
Whack>eth dev 4
Selected ethernet device 4 at Memory 91200000

Whack>net addr x.x.x.x
My address is x.x.x.x
Whack>net netmask x.x.x.x
Network mask x.x.x.x
Whack>net gateway x.x.x.x
Gateway address x.x.x.x
Whack>net server x.x.x.x
Server address x.x.x.x
Whack>boot net install cr=2 ipaddr=x.x.x.x nm=x.x.x.x rp=x.x.x.x0::rescue hn=XXXXXXXXXX_2 eth=eth4

If you see the output above, skip to step 11.
If you do not see the above output, you’ll need to halt the node as it boots by pressing Ctrl+w when you see the banner below:

+--------------------------------------------------------------------------------+
HP SPI Image 04.08.38. Release version. 17:01:11 Jul 17 2015.
+--------------------------------------------------------------------------------+
| CPU 1 x 2.40 GHz Ivy Bridge deca core HT dual
| Control Cache Size 63.100 GB (cpu mem type 11) CL6
| Set0 CC DIMM0.0 (J18000): 32758 MB CL5/9
| Set0 CC DIMM1.0 (J19000): 32758 MB CL5/9
| Data Cache Size 32 GB CL5/11
| Set0 DC DIMM0.0 (J14005): 16384 MB CL5/11
| Set0 CC DIMM1.0 (J16005): 16384 MB CL5/11
| Slot ID 3 [4 Node HP 3PAR 8440 Centerplane]
| FPGA Tornado v0.2
| SATA0 Disk SanDisk DX300128A5xnEMLC 151410400082 128GB
| PCI Slot 0 Emulex LPe16002 NO DATA 2-port
| PCI Slot 1 LSI-SAS 9300-8e
| Board 920-200048.59 FXN 2014/49/Fri 00279453 34:64:a9:ea:05:54
| Cluster serial XXXXXXXXX 34:64:a9:ea:05:55
| Board reset reason ALIVE_L
| Current Time 2016-06-28 17:06:23 (UTC-4)
+------------------------------------------------------BIOS wrote initial PCIe tree to idelog
Booting from device 0, AHCI 0
Setting FSB WDT Boot Complete State.
GRUB loading.
Welcome to GRUB!  Type ctrl+w here to halt the node.

6. Once halted, you’ll see the Whack> prompt
7. Type prom edit at the prompt. It will print only one line at a time – enter through until it drops back at the prompt.
8. Verify the node ID is set to FF and the Midplane Type and node type is set to correct value:
a. StoreServ 8200: Midplane Type 1a Node Type 20
b. StoreServ 8400: Midplane Type 1b Node Type 40
c. StoreServ 8450: Midplane Type 1e Node Type 80
d. StoreServ 8440: Midplane Type 1e Node Type f0
9. Issue the following commands successively:
a. Whack> unset sys_serial_10
b. Whack> unset w19
c. Whack> ahci init
d. Whack> ahci erase OR ide erase
e. Whack> reset
10. Node rescue should start automatically during the next post (you should see the output from step 5.
a. If node rescue fails for any reason, contact support for assistance. Note: if DNS is configured on the array and the DNS address is not pingable, rescue will fail.
11. Once the replacement node is booted fully and has joined the cluster (indicated by flashing green LED at 1hz), issue a shownode command to verify all nodes are present and healthy
12. Issue a checkhealth -svc -detail and note any issues.

I recently replaced a 7400c controller without even touching the serial port. The controller I got came in new.
I replaced the controller ( did all the safe remove steps ). First time the controller showed a node failure on the new node.
I then issued a controller to controller rescue. I forgot to connect the mgmt port to the same L2 network as the rescue node..
Then the node showed the safe removal led code. I reseated te controller and connected it to the network. I canceled the old node rescue and issued the node rescue again.

After a while the node booted and I connected the PCI-E cables, and the node joined the cluster.