Update failed 3par OS 3.2.1MU5 to 3.2.2MU6

[gnomba]

Hi guys!
Help me to solve problem with update. Previously I never used any data storage systems.
I have 3par 7400 INFORM OS 3.2.1.426 (MU5) patch P49,P53,P55.
I installed Physical SP-4.4.0.GA-142 (MU9) for update to INFORM OS 3.2.2.709 (MU6).
First try:

Code: Select all

No InSplore running 
Terminated InSplores 
No InSplore running 
Release version 3.2.1.426 (MU5)
Patches  P49,P53,P55

Component Name                   Version                         
CLI Server                       3.2.1.483 (P53)                 
CLI Client                       3.2.1.483                       
System Manager                   3.2.1.488 (P55)                 
Kernel                           3.2.1.426 (MU5)                 
TPD Kernel Code                  3.2.1.426 (MU5)                 
TPD Kernel Patch                 3.2.1.488 (P55)                  
Selected upgrade type  simple 

MESSAGE START State, No HPE 3PAR OS upgrade in progress
STATE0
 
StoreServ release version 3.2.1.426 + P49,P53,P55 
Beginning upgrade checks 
-- Test Name --------------------------------------------------------  RESULT  - 
Check Defrag                                                           PASSED  
Check Local                                                            PASSED  
Check PD                                                               PASSED  
Check Upgrade                                                          PASSED  
Check Upgrade Offline                                                  PASSED  
Check VV                                                               PASSED  
Check Cluster Name Exists                                              PASSED  


3par001 is in an acceptable state to perform a software upgrade 
   No InSplore currently running
Stopping spevent task for StoreServ 3par001
Stopping spcollect tasks for StoreServ 3par001
If running, stopping InSplore for StoreServ 3par001
 
 
Transferring software may be a long running operation. Do not abort the operation. 
 
Installing update 3.2.2.709-MU6 
Validating 3.2.2.709-MU6 
Enter Simple Upgrade state(0) 
auto doupdate OS-3.2.2.709-MU6 
substituting package for staged
substituting package for staged
Beginning upgrade of system software.
Extracting...
-/-/-/-/-/-/.-/-/-/-/-/-/.-/-/-               Applying new package...
Calculating checksums...
-/.-/   Performing pre-update checks...
-  Following PD(s) exceeds the chunklet failure threshold and needs corrective action 5,6,7,2
Update failed.
System is not currently undergoing an online upgrade.

System is not currently undergoing an online upgrade.
First node to be rebooted is node 1
   Which will be followed by node 0 
The upgrade has failed. Please view the messages above and contact HPE Customer Support (Error Code 1) 
Password Processing Done

Second try:

Code: Select all

No InSplore running 
Terminated InSplores 
No InSplore running 
Release version 3.2.1.426 (MU5)
Patches  P49,P53,P55

Component Name                   Version                         
CLI Server                       3.2.1.483 (P53)                 
CLI Client                       3.2.1.483                       
System Manager                   3.2.1.488 (P55)                 
Kernel                           3.2.1.426 (MU5)                 
TPD Kernel Code                  3.2.1.426 (MU5)                 
TPD Kernel Patch                 3.2.1.488 (P55)                  
Selected upgrade type  simple 

MESSAGE START State, No HPE 3PAR OS upgrade in progress
STATE0
 
StoreServ release version 3.2.1.426 + P49,P53,P55 
Beginning upgrade checks 
-- Test Name --------------------------------------------------------  RESULT  - 
Check Defrag                                                           PASSED  
Check Local                                                            PASSED  
Check PD                                                               PASSED  
Check Upgrade                                                          PASSED  
Check Upgrade Offline                                                  PASSED  
Check VV                                                               PASSED  
Check Cluster Name Exists                                              PASSED  


3par001 is in an acceptable state to perform a software upgrade 
   No InSplore currently running
Stopping spevent task for StoreServ 3par001
Stopping spcollect tasks for StoreServ 3par001
If running, stopping InSplore for StoreServ 3par001
 
 
Transferring software may be a long running operation. Do not abort the operation. 
 
Installing update 3.2.2.709-MU6 
Validating 3.2.2.709-MU6 
Enter Simple Upgrade state(0) 
auto doupdate OS-3.2.2.709-MU6 


System is not currently undergoing an online upgrade.
First node to be rebooted is node 1
   Which will be followed by node 0 
The upgrade has failed. Please view the messages above and contact HPE Customer Support (Error Code 1) 
Password Processing Done 
 
system-wide certificate already exists 
 
Beginning post upgrade checks 
HPE 3PAR OS version(altroot) = 3.2.1.292+P17,P18,P19,P23,P30,P50,P52, rc(0) 
-- Test Name --------------------------------------------------------  RESULT  - 
Check Defrag                                                           PASSED  
Health Check                                                           PASSED  


Post upgrade checks are passed but software upgrade failed on 3par001 due to incompatible state 
Starting spcollect tasks for StoreServ 3par001
Starting spevent task for StoreServ 3par001
 
There was an issue upgrading your StoreServ. Please contact HPE Customer Service with the output
from the above operation to resolve this problem

[koopa79]

what do you get from a checkhealth -svc -detail

The first screenshot suggests you have 4 physical disks with issues. I'm assuming they're not actually failed disks? Might need to fix those disks first.

maybe try a showpd -degraded -failed

could also try a showpd -c and it will show you how many failed chunklets on all disks.

[gnomba]

checkhealth -svc -detail:

Code: Select all

Checking alert
Checking ao
Checking cabling
Checking cage
Checking cert
Checking dar
Checking date
Checking file
Checking fs
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking pdch
Checking port
Checking qos
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Checking sp
Component -------------------Description-------------------- Qty
Alert     New alerts                                           2
Node      Filesystems which are over 90% full                  1
Node      Filesystems which are over 80% full                  1
Host      Host ports not configured for virtual port support   1
License   Licenses which have expired                          1
pdch      LDs with chunklets on a remote disk                 14
RC        Structure Incompatibility                            1
vlun      Hosts not connected to a port                        2

Component ---Identifier--- -------------------------------------------------------------Description--------------------------------------------------------------
Alert     sw_os            An Update is Available                                                                                                                
Alert     sw_os            Update OS-3.2.2.709-MU6 has failed:   Following PD(s) exceeds the chunklet failure threshold and needs corrective action: 5,6,7,2-----
Node      node:0           Filesystem /dev/sda7 mounted on "/common" is over 90% full                                                                            
Node      node:1           Filesystem /dev/sda7 mounted on "/common" is over 80% full                                                                            
Host      Port:1:1:1       Port WWN not found on FC Fabric attached to Port:0:1:1                                                                                
License   Peer Motion      License has expired                                                                                                                   
pdch      ld:worker0.usr.3 LD has 2 remote chunklets                                                                                                             
pdch      ld:worker0.usr.1 LD has 3 remote chunklets                                                                                                             
pdch      ld:admin.usr.2   LD has 2 remote chunklets                                                                                                             
pdch      ld:master0.usr.5 LD has 16 remote chunklets                                                                                                            
pdch      ld:master1.usr.5 LD has 16 remote chunklets                                                                                                            
pdch      ld:.srdata.usr.1 LD has 13 remote chunklets                                                                                                            
pdch      ld:master2.usr.5 LD has 16 remote chunklets                                                                                                            
pdch      ld:master0.usr.3 LD has 2 remote chunklets                                                                                                             
pdch      ld:worker0.usr.5 LD has 16 remote chunklets                                                                                                            
pdch      ld:master0.usr.1 LD has 3 remote chunklets                                                                                                             
pdch      ld:master1.usr.1 LD has 4 remote chunklets                                                                                                             
pdch      ld:master1.usr.3 LD has 2 remote chunklets                                                                                                             
pdch      ld:master2.usr.3 LD has 2 remote chunklets                                                                                                             
pdch      ld:master2.usr.1 LD has 4 remote chunklets                                                                                                             
RC        Structure        Remote Copy internal structure is incompatible.                                                                                       
vlun      host:master0     Host iscsi_name:iqn.1994-05.com.redhat:3443780cbb is not connected to a port                                                          
vlun      host:master0     Host iscsi_name:iqn.1994-05.com.redhat:bfbbd4d9d631 is not connected to a port

showpd -degraded -failed:

Code: Select all

No PDs listed

showpd -c:

Code: Select all

                             ------- Normal Chunklets -------- ---- Spare Chunklets ----
                             - Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State  Total  OK Fail Free Uninit Unavail Fail OK  Fail Free Uninit Fail
 0 0:0:0   SSD  normal  1787 120    0 1220      0       0    0  0     0  447      0    0
 1 0:1:0   SSD  normal  1787 159    0 1181      0       0    0  0     0  447      0    0
 2 0:2:0   SSD  normal  1787   0    0 1333      0       0    7  0     0  447      0    0
 3 0:3:0   SSD  normal  1787 131    0 1204      0       0    5  0     0  447      0    0
 4 0:4:0   SSD  normal  1787 120    0 1215      0       0    5  0     0  447      0    0
 5 0:5:0   SSD  normal  1787 131    0 1202      0       0    7  0     0  447      0    0
 6 0:6:0   SSD  normal  1787 119    0 1215      0       0    7  0     0  446      0    0
 7 0:7:0   SSD  normal  1787 167    0 1167      0       0    7  0     0  446      0    0
----------------------------------------------------------------------------------------
 8 total               14296 947    0 9737      0       0   38  0     0 3574      0    0

[gnomba]

I tryed fix failed chunklets on disk 2 in the following way:
1. connected disk to server
2. reformatted to 512 block size
3. blkdiscard
4. reformatted to 520 block size
5. returned disk to 3par

But this did not solve the problem with failed chunklets.

[koopa79]

so at least you can see the reason the upgrade failed is those PD's as per
Alert sw_os Update OS-3.2.2.709-MU6 has failed: Following PD(s) exceeds the chunklet failure threshold and needs corrective action: 5,6,7,2-----

Now I've never thought about taking a disk out of 3par and connecting it to a server and formatting it. But i know the 3pars have a database of failed disks, my worry is now you've removed it I think its failed and has now marked that disk as bad. Because in the last screen shot, i don't like the OK column shows 0 on disk 2.

I'm assuming this 3par isn't under support anymore? and that's why you haven't asked HPE to swap the disk.

run another showpd -c and see if the details on disk 2 have changed anymore? i don't know if it has to rescan the chunklets again.

[gnomba]

Yes, unfortunately there is no support for this storage.
Before disconnecting disk 2, I manually transferred all used chunklets, but after returning disk 2, they apparently did not come back automatically, so the value in the OK column is 0.
`showpd -c`:

Code: Select all

                             ------- Normal Chunklets -------- ---- Spare Chunklets ----
                             - Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State  Total  OK Fail Free Uninit Unavail Fail OK  Fail Free Uninit Fail
 0 0:0:0   SSD  normal  1787 120    0 1220      0       0    0  0     0  447      0    0
 1 0:1:0   SSD  normal  1787 159    0 1181      0       0    0  0     0  447      0    0
 2 0:2:0   SSD  normal  1787   0    0 1333      0       0    7  0     0  447      0    0
 3 0:3:0   SSD  normal  1787 131    0 1204      0       0    5  0     0  447      0    0
 4 0:4:0   SSD  normal  1787 120    0 1215      0       0    5  0     0  447      0    0
 5 0:5:0   SSD  normal  1787 131    0 1202      0       0    7  0     0  447      0    0
 6 0:6:0   SSD  normal  1787 119    0 1215      0       0    7  0     0  446      0    0
 7 0:7:0   SSD  normal  1787 167    0 1167      0       0    7  0     0  446      0    0
----------------------------------------------------------------------------------------
 8 total               14296 947    0 9737      0       0   38  0     0 3574      0    0

[gnomba]

Maybe I can somehow indicate not to use these failed chunklets or raise this threshold, for example, to 10?

[gnomba]

I recently found out that I can unmap failed chunklets and update 3parOS. Only it is not clear how to do it.

[MammaGutt]

Are you using the upgrade tool?

[gnomba]

I tried update via Service Processor (WebUI).