Vvol use >100% and Space allocation failed write protect

[feldsam]

Hello, today I experienced problem with one vvol. We use KVM virtualization on Oracle Linux 7. VM gets into pused state and in dmesg I found this "Space allocation failed write protect" on exported lun slaves.

I resumed VM and it is working fine, but I investigate futher and found that 3PR reports that vvol used more that 100% space!

Code: Select all

3PAR8450 cli% showvv -s feldcloud.one.89.vv
                                              ----------Snp----------- ------------Usr------------ -----------Total-----------                 
                                              ---(MiB)--- -(% VSize)-- ----(MiB)---- --(% VSize)-- -----------(MiB)----------- ---Efficiency---
 Id Name                Prov Compr Dedup Type  Rsvd  Used Used Wrn Lim   Rsvd   Used  Used Wrn Lim   Rsvd   Used HostWr  VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes   Yes   base 16384 13375  3.7   0   0 366720 358775 100.1   0   0 383104 372149 249942 358400    1.93     1.14
-----------------------------------------------------------------------------------------------------------------------------------------------
  1 total                                     16384 13375              366720 358775               383104 372149 249942 358400    

I also tried to issue "fstrim -av" command in the VM, but only just "Host written" field dropped to real value.

Any advice on this? Thank you!

[MammaGutt]

I'm making an educated guess that you are using tdvv2.

For tdvv2, the usr used and usr rsvd are weighted number based on data stored in DDS and data stored in DDC.

Based on those numbers I assume that your DDS is quite big and you're getting bad dedupe.

With tdvv3, those number only contain the DDC numbers.

[feldsam]

Hello, thank you for reply. It is 8450 system deployed to the production with 3.3.1 MU3, so it should be tdvv v3? How I can check?

EDIT: CPG is using version 3

Code: Select all

3PAR8450 cli% showcpg -d
                        ----Volumes---- -Usage- ------------(MiB)------------- --LD--- -RC_Usage- -Shared-
Id Name           Warn% VVs TPVVs TDVVs Usr Snp    Base     Snp   Free   Total Usr Snp  Usr   Snp  Version
 0 SSD_r1             -   0     0     0   0   0       0       0      0       0   0   0    0     0        -
 1 SSD_r5             -   0     0     0   0   0       0       0      0       0   0   0    0     0        -
 2 SSD_r6             - 569     0   568 569 568 7840256 1165952  62336 9068544   0  37    0     0        3
 4 SSD_r6_windows     -   6     0     5   6   5   91136   10240  46080  147456   0   4    0     0        3
----------------------------------------------------------------------------------------------------------
 4 total                                575 573 7931392 1176192 108416 9216000   0  41    0     0         

EDIT 2:

Total used raw capacity is 75%, it is a problem? I think that there is still 25% free for allocations. All other VVs are without problems, only this one. Same problem hapens today also.

EDIT 3:

Something happens in 3PAR and space allocations are diferent regardless of real usage in the VM

Code: Select all

3PAR8450 cli% showvv -s feldcloud.one.89.vv
                                              ----------Snp----------- -----------Usr------------ -----------Total-----------                 
                                              ---(MiB)--- -(% VSize)-- ----(MiB)---- -(% VSize)-- -----------(MiB)----------- ---Efficiency---
 Id Name                Prov Compr Dedup Type  Rsvd  Used Used Wrn Lim   Rsvd   Used Used Wrn Lim   Rsvd   Used HostWr  VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes   Yes   base 39936 39936 11.1   0   0 366720 293505 81.9   0   0 406656 333441 260737 358400    1.07     1.15
----------------------------------------------------------------------------------------------------------------------------------------------
  1 total                                     39936 39936              366720 293505              406656 333441 260737 358400

[MammaGutt]

Hehe. I'm impressed

I would probably log a support case and ask how it is possible for Usr Used to be higher that VSize on a non-tdvv2 volume

The only thing I could possibly think of would be some garbadge collection or something to that extent where data is recorded to no longer be in use while a new write isn't allowed to overwrite. At that point the data might still be considered a part of the volume, but probably not "assigned" to a "block number" or something. However it seems far-fetched and I'm just making wild guesses here.

If you'ld seen this on a "plain" thin volume, I would have been surprised... Big time.

[feldsam]

hi, looks like it is related to fstrim and garbage collection. We recently added fstrim to our backup script, so each VM do it before backup. Then this happens on two bigger vvols from about 400. Yesterday, when it magically goes back I see defragmentation jobs in system activity running.

I solved issue by creating physical copy of affected VV.

I am going to write to the HPE about this, I am on latest 3.3.1 MU5 with all patches