3par 7400 does not initialise chunklets

[dremesov]

Hey everybody

I have a strange problem with 3par 7400 2-node system
I had to replace a node on it and wanted to start from scratch so I performed deinstallation

So now it is not able to perform OOTB fully, specifically it does not create admin volume

It sees both nodes, the cage and cabling tests are passing, it sees my 4 disks
admitpd admits them and it says all 4 are valid
but next it is supposed to start chunklets initialization and this does not happen
it can wait forever saying that 20 chunklets for admin volume are not clean yet

showpd -c then says all chunklets are uninitialized
OOTB stress test log also shows that no chunklets are initialized during initialization stage

Code: Select all

root@9277883-1 Mon Oct 10 16:29:46:/# showtask -d 11792
   Id Type        Name        Status Phase Step -------StartTime------- -FinishTime- -Priority- ---User----
11792 system_task OOTB_stress active   1/1  0/1 2022-10-10 14:51:00 MSK -            n/a        sys:3parsys

Detailed status:
2022-10-10 14:51:00 MSK Created     task.
2022-10-10 14:51:00 MSK Updated     Executing "OOTB_stress" as 0:14060
2022-10-10 14:51:01 MSK Updated     Admitting new PDs...Checking for drive table upgrade packages.
2022-10-10 14:51:01 MSK Updated     Package check completed
2022-10-10 14:51:02 MSK Updated     0 disks admitted
2022-10-10 14:51:02 MSK Updated     
2022-10-10 14:51:03 MSK Updated     Waiting for chunklet init to complete
2022-10-10 14:56:04 MSK Updated     7148 chunklets to clean or 54:59 to go
2022-10-10 15:01:06 MSK Updated     7148 chunklets to clean or 49:58 to go
2022-10-10 15:06:07 MSK Updated     7148 chunklets to clean or 44:57 to go
2022-10-10 15:11:08 MSK Updated     7148 chunklets to clean or 39:55 to go
2022-10-10 15:16:10 MSK Updated     7148 chunklets to clean or 34:54 to go
2022-10-10 15:21:11 MSK Updated     7148 chunklets to clean or 29:53 to go
2022-10-10 15:26:12 MSK Updated     7148 chunklets to clean or 24:51 to go
2022-10-10 15:31:13 MSK Updated     7148 chunklets to clean or 19:50 to go
2022-10-10 15:36:15 MSK Updated     7148 chunklets to clean or 14:49 to go
2022-10-10 15:41:16 MSK Updated     7148 chunklets to clean or 9:48 to go
2022-10-10 15:46:17 MSK Updated     7148 chunklets to clean or 4:46 to go
2022-10-10 15:51:19 MSK Updated     7148 chunklets to clean or 0:-15 to go
2022-10-10 15:51:19 MSK Updated     Waiting for battery tests to complete
2022-10-10 15:52:25 MSK Updated     Starting stress on node 0
2022-10-10 15:52:26 MSK Updated     Starting stress on node 1
2022-10-10 15:52:27 MSK Updated     Stress started, waiting for stress completion.

root@9277883-1 Mon Oct 10 16:29:48:/# showpd -c
                             ------- Normal Chunklets -------- ---- Spare Chunklets ----
                             - Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State  Total OK  Fail Free Uninit Unavail Fail OK  Fail Free Uninit Fail
 0 0:0:0   SSD  normal  1787  0     0    0   1787       0    0  0     0    0      0    0
 1 0:1:0   SSD  normal  1787  0     0    0   1787       0    0  0     0    0      0    0
 2 0:2:0   SSD  normal  1787  0     0    0   1787       0    0  0     0    0      0    0
 3 0:3:0   SSD  normal  1787  0     0    0   1787       0    0  0     0    0      0    0
----------------------------------------------------------------------------------------
 4 total                7148  0     0    0   7148       0    0  0     0    0      0    0
root@9277883-1 Mon Oct 10 16:30:38:/# 

Code: Select all

root@9277883-0 Mon Oct 10 18:53:13:~# showversion
Release version 3.3.1 (MU3)
Patches:  P58

Component Name                   Version                         
CLI Server                       3.3.1 (P58)                     
CLI Client                       3.3.1                           
System Manager                   3.3.1 (P58)                     
Kernel                           3.3.1 (MU3)                     
TPD Kernel Code                  3.3.1 (MU3)                     
TPD Kernel Patch                 3.3.1 (P58)                     

I see no events regarding this
just see that cage is constantly changing its state
showpdch -cng says all chunklets have to be cleaned but nothing happens, the system cleans nothing

Does anybody know where to dig next or how to trigger chunklets initialization manually?

[MammaGutt]

Pretty sure you need at least 6 disks in a 3PAR.

4 is probably not enough.

[dremesov]

Pretty sure you need at least 6 disks in a 3PAR.

4 is probably not enough.

Thanks for the advice but adding disks does not help, just added another 4 and it still does nothing, does not initialize anything

[MammaGutt]

Pretty sure you need at least 6 disks in a 3PAR.

4 is probably not enough.

Thanks for the advice but adding disks does not help, just added another 4 and it still does nothing, does not initialize anything

Okay, so now you have 8x SSDs, that should be enough. Did you do back in the box, out of the box again?

What type of SSDs do you have? (showpd -i)?

Side note…. Your system has a very strange serial number. In legacy 3PAR each generation had it’s own range of serials… V-class/10000 started with 14 and 7000 started with 16. Yours start with 92…..

[dremesov]

Yes, I restarted the OOTB after adding disks and it still did not initialize chunklets

Finally I managed to resolve the issue by rebooting to 3.2.2 which I had on altroot
after admithw in 3.2.2 it worked and started initialization process
Then I rebooted to 3.3.1 (MU3) once again and it continued to initialize chunklets properly
Then I was able to update to 3.3.1 (MU5) and with this version it wipes and runs OOTB normally
I guess my 3.3.1 (MU3) was somehow upgraded incorrectly or has a bug
Patched MU5 seems working just fine

As for SN - yes, it has that number printed on a label on the cage