Page 1 of 1
Vvol use >100% and Space allocation failed write protect
Posted: Tue Jan 19, 2021 6:52 pm
by feldsam
Hello, today I experienced problem with one vvol. We use KVM virtualization on Oracle Linux 7. VM gets into pused state and in dmesg I found this "Space allocation failed write protect" on exported lun slaves.
I resumed VM and it is working fine, but I investigate futher and found that 3PR reports that vvol used more that 100% space!
Code: Select all
3PAR8450 cli% showvv -s feldcloud.one.89.vv
----------Snp----------- ------------Usr------------ -----------Total-----------
---(MiB)--- -(% VSize)-- ----(MiB)---- --(% VSize)-- -----------(MiB)----------- ---Efficiency---
Id Name Prov Compr Dedup Type Rsvd Used Used Wrn Lim Rsvd Used Used Wrn Lim Rsvd Used HostWr VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes Yes base 16384 13375 3.7 0 0 366720 358775 100.1 0 0 383104 372149 249942 358400 1.93 1.14
-----------------------------------------------------------------------------------------------------------------------------------------------
1 total 16384 13375 366720 358775 383104 372149 249942 358400
I also tried to issue "fstrim -av" command in the VM, but only just "Host written" field dropped to real value.
Any advice on this? Thank you!
Re: Vvol use >100% and Space allocation failed write protect
Posted: Wed Jan 20, 2021 1:25 am
by MammaGutt
I'm making an educated guess that you are using tdvv2.
For tdvv2, the usr used and usr rsvd are weighted number based on data stored in DDS and data stored in DDC.
Based on those numbers I assume that your DDS is quite big and you're getting bad dedupe.
With tdvv3, those number only contain the DDC numbers.
Re: Vvol use >100% and Space allocation failed write protect
Posted: Wed Jan 20, 2021 5:17 am
by feldsam
Hello, thank you for reply. It is 8450 system deployed to the production with 3.3.1 MU3, so it should be tdvv v3? How I can check?
EDIT: CPG is using version 3
Code: Select all
3PAR8450 cli% showcpg -d
----Volumes---- -Usage- ------------(MiB)------------- --LD--- -RC_Usage- -Shared-
Id Name Warn% VVs TPVVs TDVVs Usr Snp Base Snp Free Total Usr Snp Usr Snp Version
0 SSD_r1 - 0 0 0 0 0 0 0 0 0 0 0 0 0 -
1 SSD_r5 - 0 0 0 0 0 0 0 0 0 0 0 0 0 -
2 SSD_r6 - 569 0 568 569 568 7840256 1165952 62336 9068544 0 37 0 0 3
4 SSD_r6_windows - 6 0 5 6 5 91136 10240 46080 147456 0 4 0 0 3
----------------------------------------------------------------------------------------------------------
4 total 575 573 7931392 1176192 108416 9216000 0 41 0 0
EDIT 2:
Total used raw capacity is 75%, it is a problem? I think that there is still 25% free for allocations. All other VVs are without problems, only this one. Same problem hapens today also.
EDIT 3:
Something happens in 3PAR and space allocations are diferent regardless of real usage in the VM
Code: Select all
3PAR8450 cli% showvv -s feldcloud.one.89.vv
----------Snp----------- -----------Usr------------ -----------Total-----------
---(MiB)--- -(% VSize)-- ----(MiB)---- -(% VSize)-- -----------(MiB)----------- ---Efficiency---
Id Name Prov Compr Dedup Type Rsvd Used Used Wrn Lim Rsvd Used Used Wrn Lim Rsvd Used HostWr VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes Yes base 39936 39936 11.1 0 0 366720 293505 81.9 0 0 406656 333441 260737 358400 1.07 1.15
----------------------------------------------------------------------------------------------------------------------------------------------
1 total 39936 39936 366720 293505 406656 333441 260737 358400
Re: Vvol use >100% and Space allocation failed write protect
Posted: Thu Jan 21, 2021 5:50 am
by MammaGutt
Hehe. I'm impressed
I would probably log a support case and ask how it is possible for Usr Used to be higher that VSize on a non-tdvv2 volume
The only thing I could possibly think of would be some garbadge collection or something to that extent where data is recorded to no longer be in use while a new write isn't allowed to overwrite. At that point the data might still be considered a part of the volume, but probably not "assigned" to a "block number" or something. However it seems far-fetched and I'm just making wild guesses here.
If you'ld seen this on a "plain" thin volume, I would have been surprised... Big time.
Re: Vvol use >100% and Space allocation failed write protect
Posted: Thu Jan 21, 2021 8:52 am
by feldsam
hi, looks like it is related to fstrim and garbage collection. We recently added fstrim to our backup script, so each VM do it before backup. Then this happens on two bigger vvols from about 400. Yesterday, when it magically goes back I see defragmentation jobs in system activity running.
I solved issue by creating physical copy of affected VV.
I am going to write to the HPE about this, I am on latest 3.3.1 MU5 with all patches