InServ E200 more than two disk drives failed

2know3par · **Joined:** Sat Feb 27, 2016 2:24 am **Posts:** 18

Help!

I have an old E200 3par system that is off support. System has one SP, one nod controller and two Disk Cages.
Each disk cage has 16 FC 400GB disk drives. There no umber lights in front or back, but showpd –failed shows some disks failed and showpd –I shows 43 disks total, although I have only 32 disks:

cli% showpd -failed –degraded
--Size(MB)-- ----Ports----
Id CagePos Type Speed(K) State Total Free A B
1 0:12:0? FC 10 failed 380928 0 ----- -----
3 0:3:0? FC 10 failed 380928 0 ----- -----
7 0:7:0? FC 10 failed 380928 0 ----- -----
10 0:10:0? FC 10 failed 380928 0 ----- -----
12 0:12:0? FC 10 failed 380928 0 ----- -----
15 0:15:0? FC 10 failed 380928 0 ----- -----
16 1:0:0? FC 10 failed 380928 0 ----- -----
20 1:4:0? FC 10 failed 380928 0 ----- -----
21 1:5:0? FC 10 failed 380928 0 ----- -----
25 1:9:0? FC 10 failed 380928 0 ----- -----
30 1:14:0? FC 10 failed 380928 0 ----- -----
----------------------------------------------------------
11 total 4190208 0
cli%

cli% showpd –i
Id CagePos State ----Node_WWN---- --MFR-- ---Model--- -FW_Rev-
0 0:0:0 normal 2000001D38389E2B SEAGATE ST3400755FC XRC1
1 0:12:0? failed 2000001D38389574 SEAGATE ST3400755FC XRC1
2 0:2:0 normal 2000001D38389D93 SEAGATE ST3400755FC XRC1
3 0:3:0? failed 2000001D38387B4E SEAGATE ST3400755FC XRC1
4 0:4:0 normal 2000001D38389EDF SEAGATE ST3400755FC XRC1
5 0:5:0 normal 2000001D38387AAE SEAGATE ST3400755FC XRC1
6 0:6:0 normal 2000001D38387A36 SEAGATE ST3400755FC XRC1
7 0:7:0? failed 2000001D38387B3C SEAGATE ST3400755FC XRC1
8 0:8:0 normal 2000001D38387B99 SEAGATE ST3400755FC XRC1
9 0:9:0 normal 2000001D38389C49 SEAGATE ST3400755FC XRC1
10 0:10:0? failed 2000001D383894B6 SEAGATE ST3400755FC XRC1
11 0:11:0 normal 2000001D38389F83 SEAGATE ST3400755FC XRC1
12 0:12:0? failed 2000001D38387CB4 SEAGATE ST3400755FC XRC1
13 0:13:0 normal 2000001D38389C7B SEAGATE ST3400755FC XRC1
14 0:14:0 normal 2000001D38389FAA SEAGATE ST3400755FC XRC1
15 0:15:0? failed 2000001D38389E35 SEAGATE ST3400755FC XRC1
16 1:0:0? failed 2000001D38389703 SEAGATE ST3400755FC XRC1
17 1:1:0 normal 2000001D38389ED1 SEAGATE ST3400755FC XRC1
18 1:2:0 normal 2000001D38389F59 SEAGATE ST3400755FC XRC1
19 1:3:0 normal 2000001D38387AFF SEAGATE ST3400755FC XRC1
20 1:4:0? failed 2000001D38387A4E SEAGATE ST3400755FC XRC1
21 1:5:0? failed 2000001D3838A056 SEAGATE ST3400755FC XRC1
22 1:6:0 normal 2000001D38389F93 SEAGATE ST3400755FC XRC1
23 1:7:0 normal 2000001D38389EDE SEAGATE ST3400755FC XRC1
24 1:8:0 normal 2000001D38389CA9 SEAGATE ST3400755FC XRC1
25 1:9:0? failed 2000001D38387C17 SEAGATE ST3400755FC XRC1
26 1:10:0 normal 2000001D38387208 SEAGATE ST3400755FC XRC1
27 0:1:0 normal 2000001D38AC8A45 SEAGATE ST3400755FC XRC1
28 1:12:0 normal 2000001D38389D28 SEAGATE ST3400755FC XRC1
29 1:13:0 normal 2000001D38387C82 SEAGATE ST3400755FC XRC1
30 1:14:0? failed 2000001D38389505 SEAGATE ST3400755FC XRC1
31 1:15:0 normal 2000001D38387CD9 SEAGATE ST3400755FC XRC1
32 1:11:0 normal 2000001D38A3DA73 SEAGATE ST3400755FC XRC1
34 0:12:0 normal 20000024B636CE67 SEAGATE ST3400755FC XRC1
35 1:4:0 normal 2000001D38AC8B7D SEAGATE ST3400755FC XRC1
36 1:5:0 normal 2000001D38AC6930 SEAGATE ST3400755FC XRC1
37 1:9:0 normal 2000001D38AC8CD2 SEAGATE ST3400755FC XRC1
38 0:3:0 normal 2000001D38387046 SEAGATE ST3400755FC XRC1
39 0:7:0 normal 2000001D38386FA0 SEAGATE ST3400755FC XRC1
40 0:10:0 normal 2000001D383872E5 SEAGATE ST3400755FC XRC1
41 0:15:0 normal 2000001D38AC87D5 SEAGATE ST3400755FC XRC1
42 1:0:0 normal 2000001D38AC69B5 SEAGATE ST3400755FC XRC1
43 1:14:0 normal 2000001D38AC894C SEAGATE ST3400755FC XRC1
------------------------------------------------------------------------
43 total
cli%

MammaGutt · **Posted:** Sat Feb 27, 2016 5:30 am

Seems like old disks has not been replaced correctly and not cleaned up the old pd entry.

3 0:3:0? failed 2000001D38387B4E SEAGATE ST3400755FC XRC1
38 00:03:00 normal 2000001D38387046 SEAGATE ST3400755FC XRC1
7 0:7:0? failed 2000001D38387B3C SEAGATE ST3400755FC XRC1
39 00:07:00 normal 2000001D38386FA0 SEAGATE ST3400755FC XRC1
10 0:10:0? failed 2000001D383894B6 SEAGATE ST3400755FC XRC1
40 00:10:00 normal 2000001D383872E5 SEAGATE ST3400755FC XRC1
1 0:12:0? failed 2000001D38389574 SEAGATE ST3400755FC XRC1
12 0:12:0? failed 2000001D38387CB4 SEAGATE ST3400755FC XRC1
34 00:12:00 normal 20000024B636CE67 SEAGATE ST3400755FC XRC1
15 0:15:0? failed 2000001D38389E35 SEAGATE ST3400755FC XRC1
41 00:15:00 normal 2000001D38AC87D5 SEAGATE ST3400755FC XRC1
16 1:0:0? failed 2000001D38389703 SEAGATE ST3400755FC XRC1
42 01:00:00 normal 2000001D38AC69B5 SEAGATE ST3400755FC XRC1
20 1:4:0? failed 2000001D38387A4E SEAGATE ST3400755FC XRC1
35 01:04:00 normal 2000001D38AC8B7D SEAGATE ST3400755FC XRC1
21 1:5:0? failed 2000001D3838A056 SEAGATE ST3400755FC XRC1
36 01:05:00 normal 2000001D38AC6930 SEAGATE ST3400755FC XRC1
25 1:9:0? failed 2000001D38387C17 SEAGATE ST3400755FC XRC1
37 01:09:00 normal 2000001D38AC8CD2 SEAGATE ST3400755FC XRC1
30 1:14:0? failed 2000001D38389505 SEAGATE ST3400755FC XRC1
43 01:14:00 normal 2000001D38AC894C SEAGATE ST3400755FC XRC1

Richard Siemers · **Posted:** Sat Feb 27, 2016 7:20 am

See how you have duplicates, like 0:10:0, the first one has a ? by by it, and the current PDID is toward the bottom of the list.

After a failed drive is replaced, i believe "dismisspd" is used against the old PDID.

2know3par · **Joined:** Sat Feb 27, 2016 2:24 am **Posts:** 18

I know that the disks were replaced without proper order, that’s why I am here 
And, yes, it would be too easy if dismispd would work, but it doesn’t 
Any other suggestions? Thanks

Richard Siemers · **Posted:** Sat Feb 27, 2016 5:27 pm

When you try the dismisspd, what error do you get?

Does "servicemag status" give you anything back?

2know3par · **Joined:** Sat Feb 27, 2016 2:24 am **Posts:** 18

Sorry, I should have included them on the first place. I was messing with servicemag command and cleared those messages now. But this is what was originally when I first time logged to the machine:

cli% dismisspd 1 Error : Pd id 1 is in use
cli% dismisspd 30 Error : Pd id 30 is in use
cli% dismisspd 25 Error : Pd id 25 is in use
cli% dismisspd 21 Error : Pd id 21 is in use
cli% dismisspd 21 Error : Pd id 20 is in use
cli% dismisspd 20 Error : Pd id 16 is in use
cli% dismisspd 16 Error : Pd id 15 is in use
cli% dismisspd 15 Error : Pd id 12 is referenced by chunklet 1:1420
cli% dismisspd 12 Error : Pd id 10 is in use
cli% dismisspd 10 Error : Pd id 7 is in use
cli% dismisspd 7 Error : Pd id 3 is in use
cli% dismisspd 30 Error : Pd id 3 is in use
cli% dismisspd 31Error : Pd id 1 is in use

cli% servicemag status
Cage 0, magazine 3:
A servicemag start command failed on this magazine.
The command completed at Mon Feb 22 10:07:14 2016.
Failed -- -- Failed
cli%

cli% servicemag status
Cage 0, magazine 3:
A servicemag resume command failed on this magazine.
The command completed at Mon Jun 30 18:03:28 2014.
servicemag resume 0 3 -- Failed

Cage 0, magazine 12:
A servicemag start command failed on this magazine.
The command completed at Wed Feb 17 07:56:24 2016.
servicemag start -pdid 1 -- Failed

Cage 1, magazine 4:
A servicemag start command failed on this magazine.
The command completed at Mon Jun 2 08:37:57 2014.
servicemag start 1 4 -- Failed

Cage 1, magazine 5:
A servicemag start command failed on this magazine.
The command completed at Mon Jun 2 08:38:18 2014.
servicemag start 1 5 -- Failed

Cage 1, magazine 9:
A servicemag start command failed on this magazine.
The command completed at Mon Jun 2 08:38:39 2014.
servicemag start 1 9 -- Failed
cli%

Richard Siemers · **Posted:** Sat Feb 27, 2016 10:53 pm

Glad to hear you got it cleared out. So despite all those disk fails and service mag issues, was there any data loss or did the system deal with it?

2know3par · **Joined:** Sat Feb 27, 2016 2:24 am **Posts:** 18

This is development machine, and it’s in a cluster, no data was lost.
Now system sees all disks space aroung 12TB, and even more… like those 11 disks it shows as failed volume (around 4TB), and that’s what bothering development crew.

cli% showpd -state
Id CagePos Type -State- --------------------------Detailed_State--------------------------
0 0:0:0 FC normal normal
1 0:12:0? FC failed prolonged_missing,no_valid_ports,not_available_for_allocations
2 0:2:0 FC normal normal
3 0:3:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
4 0:4:0 FC normal normal
5 0:5:0 FC normal normal
6 0:6:0 FC normal normal
7 0:7:0? FC failed invalid_media,no_valid_ports,missing
8 0:8:0 FC normal normal
9 0:9:0 FC normal normal
10 0:10:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
11 0:11:0 FC normal normal
12 0:12:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
13 0:13:0 FC normal normal
14 0:14:0 FC normal normal
15 0:15:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
16 1:0:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
17 1:1:0 FC normal normal
18 1:2:0 FC normal normal
19 1:3:0 FC normal normal
20 1:4:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
21 1:5:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
22 1:6:0 FC normal normal
23 1:7:0 FC normal normal
24 1:8:0 FC normal normal
25 1:9:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
26 1:10:0 FC normal normal
27 0:1:0 FC normal normal
28 1:12:0 FC normal normal
29 1:13:0 FC normal normal
30 1:14:0? FC failed invalid_media,no_valid_ports,missing,not_available_for_allocations
31 1:15:0 FC normal normal
32 1:11:0 FC normal normal
33 0:3:0? FC failed prolonged_missing,no_valid_ports,not_available_for_allocations
34 0:12:0 FC normal servicing
35 1:4:0 FC normal servicing
36 1:5:0 FC normal servicing
37 1:9:0 FC normal servicing
38 0:3:0 FC normal servicing
39 0:7:0 FC normal servicing
40 0:10:0 FC normal servicing
41 0:15:0 FC normal servicing
42 1:0:0 FC normal servicing
43 1:14:0 FC normal servicing
------------------------------------------------------------------------------------------
44 total

2know3par · **Joined:** Sat Feb 27, 2016 2:24 am **Posts:** 18

"Glad to hear you got it cleared out. So despite all those disk fails and service mag issues, was there any data loss or did the system deal with it?" - I cleared servicemag error messages, but problem still there. It still shows 11 extra disks and they are obviously showing as a failed drives.

Richard Siemers · **Posted:** Sat Feb 27, 2016 11:54 pm

Hrmm, I am guessing that since your servicemags didn't run through correctly, that you probably have a ton of chunklets residing in spare space but still attributed to belonging to the original PDID.

The showpdch command, with the option to show you which chunklets gave moved, should help to show you which chunklets are not in their proper spot. I'll have to defer back to the community as to what method would be best to put them them back where the belong. Servicemag normal handles this whole process for you, but I am sure it can be done manually in a pinch.

HPE Storage Users Group

InServ E200 more than two disk drives failed

Who is online