CPG filled up, how do I recover the space?

[Subversive]

Hi, new 3par user here. We had a CPG fill up tonight (I think the zero detection is not working either, but that's another story for another day). Anyway, I deleted the Datastore on VMware, then deleted the VV from the 3Par. The 3Par is showing the free space now (approx 3TB), however the copy space still appears to be full and I don't know how to reclaim it. I'm on the line with HP support, but they are not exactly inspiring me with confidence. Can anyone advise on how I would do this?

3par.PNG (35.53 KiB) Viewed 36304 times

[Subversive]

So, even though the 'compact CPG' command was greyed out from the GUI, I was able to run compactcpg from the CLI and it ran succesfully. However, my CPG is still showing in a failed state. I've been on the line with HP for nearly 7 hours and they can't seem to tell me anything useful, I even had to find the CLI command myself! Anyone know how to bring the CPG back online?

3par2.PNG (5.16 KiB) Viewed 36300 times

[kte]

It seems like it is you're copy space that can't do an extend, do you use thin or full provisioning?
Do you need snapshots? maybe disable the snapshot on you're luns or delete some snapshots.

K

[Architect]

Copy space is by default used for tpvv volumes.
You can see that the compactcpg had done its job as the cpg is now the same size as the storage in use (shrunk from 11 to 8TB).

Did you go to the alerts tab and mark the alert there as "Fixed"?

Besides that, My guess is the 3par was forced to create some LDs with serious degraded parameters, this might need some fixing as well

Could you post or PM me the output of the following commands, ran on cli?:

- checkupgrade
- showalert
- showcpg -sdg
- showcpg -sag
- showcpg -alert
- showld -cpg FC-RAID5-SS6

[Subversive]

Hi, I did mark the alerts as fixed, however the CPG remains in a 'failed' state. However I am able to create and export VV's, so HP is suggesting this may be a false positive error in the GUI. Still not resolved as of now.

Here are the results of the various commands:

3PAR1 cli% checkupgrade
The current system configuration is valid for proceeding with an online upgrade.

3PAR1 cli% showalert
Id : 18
State : New
Message Code: 0x027000f
Time : 2013-08-30 22:10:09 MDT
Severity : Minor
Type : FC raw space allocation 75% alert
Message : Total FC raw space usage at 11183G (above 75% of total 14688G)

3PAR1 cli% showcpg -sdg
------(MB)------

Id Name Warn Limit Grow Args

0 FC-RAID5-SS6 - - 32768 -t r5 -ha mag -ssz 6 -ss 128 -ch first -p -devtype FC


3PAR1 cli% showcpg -sag
-----(MB)------
Id Name Warn Limit Grow Args
0 FC-RAID5-SS6 - - 8192 -ha mag -p -devtype FC


3PAR1 cli% showcpg -alert
------------- Data ------------- ---- Adm ----
-(MB)-- - Setting(MB) - Alerts -(MB)- Alerts
Id Name Warn% Total Warn Limit W% W L F Total W% F
0 FC-RAID5-SS6 - 8448000 - - - - - Y 16384 - -

3PAR1 cli% showld -cpg FC-RAID5-SS6
LDs for SA:
Id Name RAID -Detailed_State- Own SizeMB UsedMB Use Lgct LgId WThru MapV

11 tp-0-sa-0.0 1 normal 0/1 8192 3584 C,SA 0 --- N Y

12 tp-0-sa-0.1 1 normal 1/0 8192 3584 C,SA 0 --- N Y

--------------------------------------------------------------------------------

2 16384 7168

LDs for SD:
Id Name RAID -Detailed_State- Own SizeMB UsedMB Use Lgct LgId WThru M
apV
13 tp-0-sd-0.0 5 normal 0/1 256000 256000 C,SD 0 --- N
Y
14 tp-0-sd-0.1 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
15 tp-0-sd-0.2 5 normal 0/1 261120 261120 C,SD 0 --- N
Y
16 tp-0-sd-0.3 5 normal 1/0 256000 256000 C,SD 0 --- N
Y
17 tp-0-sd-0.4 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
18 tp-0-sd-0.6 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
19 tp-0-sd-0.5 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
20 tp-0-sd-0.8 5 normal 1/0 256000 256000 C,SD 0 --- N
Y
21 tp-0-sd-0.7 5 normal 0/1 15360 15360 C,SD 0 --- N
Y
22 tp-0-sd-0.10 5 normal 1/0 15360 15360 C,SD 0 --- N
Y
26 tp-0-sd-0.13 5 normal 0/1 15360 6272 C,SD 0 --- N
Y
30 tp-0-sd-0.18 5 normal 1/0 30720 29440 C,SD 0 --- N
Y
33 tp-0-sd-0.20 5 normal 0/1 230400 228096 C,SD 0 --- N
Y
34 tp-0-sd-0.21 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
35 tp-0-sd-0.23 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
36 tp-0-sd-0.24 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
37 tp-0-sd-0.22 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
38 tp-0-sd-0.26 5 normal 0/1 5120 5120 C,SD 0 --- N
Y
39 tp-0-sd-0.25 5 normal 0/1 261120 261120 C,SD 0 --- N
Y
40 tp-0-sd-0.28 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
41 tp-0-sd-0.29 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
42 tp-0-sd-0.27 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
43 tp-0-sd-0.30 5 normal 0/1 261120 261120 C,SD 0 --- N
Y
44 tp-0-sd-0.31 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
45 tp-0-sd-0.33 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
46 tp-0-sd-0.32 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
47 tp-0-sd-0.35 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
48 tp-0-sd-0.34 5 normal 0/1 261120 261120 C,SD 0 --- N
Y
49 tp-0-sd-0.36 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
50 tp-0-sd-0.38 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
51 tp-0-sd-0.39 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
52 tp-0-sd-0.37 5 normal 0/1 245760 245376 C,SD 0 --- N
Y
53 tp-0-sd-0.40 5 normal 0/1 261120 261120 C,SD 0 --- N
Y
54 tp-0-sd-0.42 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
55 tp-0-sd-0.43 5 normal 1/0 261120 261120 C,SD 0 --- N
Y
56 tp-0-sd-0.41 5 normal 0/1 245760 245760 C,SD 0 --- N
Y
57 tp-0-sd-0.45 5 normal 1/0 245760 245760 C,SD 0 --- N
Y
58 tp-0-sd-0.44 5 normal 0/1 92160 92160 C,SD 0 --- N
Y
59 tp-0-sd-0.46 5 normal 0/1 81920 81920 C,SD 0 --- N
Y
60 tp-0-sd-0.48 5 normal 1/0 107520 107520 C,SD 0 --- N
Y
61 tp-0-sd-0.47 5 normal 1/0 20480 20480 C,SD 0 --- N
Y
--------------------------------------------------------------------------------
---
41 8448000 8434944


Here are the alerts I have marked as fixed:

3paralerts.PNG (14.76 KiB) Viewed 36256 times

The failed error still shows the same as before:

3parfailedcpg.PNG (7.66 KiB) Viewed 36256 times

[Architect]

jup, it tried to create some LDs with degraded parameters.
The IMC might be right here, that the cpg really isn't one 100% healthy at the moment.

Mssing that info though, i've made a typo in the showld command (sorry), could you do me a favour and rerun it with "-d" added on the end?

That way I can spot the creation parameters and tell you if and which LDs to fix. I want to be certain we have an overview on all LDs.

[Subversive]

Hi, so I just marked the minor alert which was showing, about me being over 75% full, as 'fixed', and the CPG came back showing as OK now. Weird that it would keep showing a failed state with nothing but a minor alert.

Just to be thorough, here is the output of the requested command.

3PAR1 cli% showld -cpg FC-RAID5-SS6 -d
LDs for SA:
Id Name CPG RAID Own SizeMB RSizeMB RowSz StepKB SetSz Refcnt Av
ail CAvail -----CreationTime------ -CreationPattern-
11 tp-0-sa-0.0 FC-RAID5-SS6 1 0/1 8192 24576 4 256 3 0 ca
ge port 2013-06-05 10:19:12 MDT -p -devtype FC
12 tp-0-sa-0.1 FC-RAID5-SS6 1 1/0 8192 24576 4 256 3 0 ca
ge port 2013-06-05 10:19:12 MDT -p -devtype FC
--------------------------------------------------------------------------------
----------------------------------------------------
2 16384 49152

LDs for SD:
Id Name CPG RAID Own SizeMB RSizeMB RowSz StepKB SetSz Refcnt
Avail CAvail -----CreationTime------ -CreationPattern-
13 tp-0-sd-0.0 FC-RAID5-SS6 5 0/1 256000 307200 2 128 6 0
mag mag 2013-06-05 10:19:12 MDT -p -devtype FC
14 tp-0-sd-0.1 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-06-05 10:19:12 MDT -p -devtype FC
15 tp-0-sd-0.2 FC-RAID5-SS6 5 0/1 261120 313344 3 128 6 0
mag mag 2013-06-05 11:22:05 MDT -p -devtype FC
16 tp-0-sd-0.3 FC-RAID5-SS6 5 1/0 256000 307200 2 128 6 0
mag mag 2013-06-05 11:22:05 MDT -p -devtype FC
17 tp-0-sd-0.4 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-06-05 16:33:37 MDT -p -devtype FC
18 tp-0-sd-0.6 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-06-05 17:33:52 MDT -p -devtype FC
19 tp-0-sd-0.5 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-06-06 08:55:12 MDT -p -devtype FC
20 tp-0-sd-0.8 FC-RAID5-SS6 5 1/0 256000 307200 2 128 6 0
mag mag 2013-06-06 21:36:45 MDT -p -devtype FC
21 tp-0-sd-0.7 FC-RAID5-SS6 5 0/1 30720 36864 3 128 6 0
mag mag 2013-06-07 10:59:20 MDT -p -devtype FC
22 tp-0-sd-0.10 FC-RAID5-SS6 5 1/0 30720 36864 3 128 6 0
mag mag 2013-06-11 11:38:19 MDT -p -devtype FC
26 tp-0-sd-0.13 FC-RAID5-SS6 5 0/1 15360 18432 3 128 6 0
mag mag 2013-06-12 22:08:51 MDT -p -devtype FC
30 tp-0-sd-0.18 FC-RAID5-SS6 5 1/0 30720 36864 3 128 6 0
mag mag 2013-06-13 21:58:53 MDT -p -devtype FC
33 tp-0-sd-0.20 FC-RAID5-SS6 5 0/1 230400 276480 3 128 6 0
mag mag 2013-06-14 15:45:50 MDT -p -devtype FC
34 tp-0-sd-0.21 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-06-18 05:04:49 MDT -p -devtype FC
35 tp-0-sd-0.23 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-06-18 20:22:22 MDT -p -devtype FC
36 tp-0-sd-0.24 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-06-27 16:20:33 MDT -p -devtype FC
37 tp-0-sd-0.22 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-07-09 11:07:13 MDT -p -devtype FC
38 tp-0-sd-0.26 FC-RAID5-SS6 5 0/1 10240 12288 1 128 6 0
mag mag 2013-07-11 11:25:27 MDT -p -devtype FC
39 tp-0-sd-0.25 FC-RAID5-SS6 5 0/1 261120 313344 3 128 6 0
mag mag 2013-07-15 16:04:05 MDT -p -devtype FC
40 tp-0-sd-0.28 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-07-24 13:13:35 MDT -p -devtype FC
41 tp-0-sd-0.29 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-07-28 21:58:12 MDT -p -devtype FC
42 tp-0-sd-0.27 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-07-31 14:14:10 MDT -p -devtype FC
43 tp-0-sd-0.30 FC-RAID5-SS6 5 0/1 261120 313344 3 128 6 0
mag mag 2013-08-02 18:31:00 MDT -p -devtype FC
44 tp-0-sd-0.31 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-08-02 18:31:00 MDT -p -devtype FC
45 tp-0-sd-0.33 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-08-03 04:33:17 MDT -p -devtype FC
46 tp-0-sd-0.32 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-08-03 07:35:59 MDT -p -devtype FC
47 tp-0-sd-0.35 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-08-03 08:50:17 MDT -p -devtype FC
48 tp-0-sd-0.34 FC-RAID5-SS6 5 0/1 261120 313344 3 128 6 0
mag mag 2013-08-03 21:49:20 MDT -p -devtype FC
49 tp-0-sd-0.36 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-08-04 13:04:33 MDT -p -devtype FC
50 tp-0-sd-0.38 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-08-04 17:51:38 MDT -p -devtype FC
51 tp-0-sd-0.39 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-08-04 20:09:18 MDT -p -devtype FC
52 tp-0-sd-0.37 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-08-05 19:39:57 MDT -p -devtype FC
53 tp-0-sd-0.40 FC-RAID5-SS6 5 0/1 261120 313344 3 128 6 0
mag mag 2013-08-06 07:43:53 MDT -p -devtype FC
54 tp-0-sd-0.42 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-08-07 01:24:51 MDT -p -devtype FC
55 tp-0-sd-0.43 FC-RAID5-SS6 5 1/0 261120 313344 3 128 6 0
mag mag 2013-08-08 10:40:46 MDT -p -devtype FC
56 tp-0-sd-0.41 FC-RAID5-SS6 5 0/1 245760 294912 2 128 6 0
mag mag 2013-08-08 18:36:48 MDT -p -devtype FC
57 tp-0-sd-0.45 FC-RAID5-SS6 5 1/0 245760 294912 2 128 6 0
mag mag 2013-08-14 10:04:18 MDT -p -devtype FC
58 tp-0-sd-0.44 FC-RAID5-SS6 5 0/1 92160 110592 3 128 6 0
mag mag 2013-08-21 15:55:50 MDT -p -devtype FC
59 tp-0-sd-0.46 FC-RAID5-SS6 5 0/1 81920 98304 2 128 6 0
mag mag 2013-08-23 10:19:07 MDT -p -devtype FC
60 tp-0-sd-0.48 FC-RAID5-SS6 5 1/0 107520 129024 3 128 6 0
mag mag 2013-08-26 19:59:24 MDT -p -devtype FC
61 tp-0-sd-0.47 FC-RAID5-SS6 5 1/0 20480 24576 2 128 6 0
mag mag 2013-08-30 17:21:54 MDT -p -devtype FC
--------------------------------------------------------------------------------
-------------------------------------------------------
41 8483840 10180608

[Architect]

Good. Nothing wrong with the creation of the LDs, so if your vvols are healthy then you should be fine. think that the IMC was hanging on all open alerts until it reset it's state. Fixing that minor one did the trick in that case.

setting a user initiated alarm on a higher value will suppress the default raw space alerts at 50/75/85 etc
you can do that in the IMC, click first system in lower left pane, then the system you are managing, then edit. watch out though you wont be running out of space a second time.