Reclamation of thin provisioning on Unix (AIX)

[alicat13]

We are in the process of testing thin reclalmation on an AIX server. We have created a 560GB test area consisting of 16 x 35GB virtual volumes. We populated the area with junk data until usage was around 80% on both the O/S and the thin provisioning. We then deleted the use with the O/S shoowing 1% usage and the thin provisioning showing 80% as expected. I then created a file full of zero's using :
dd if=/dev/zero > /test/zero.txt
The file system reached 100% and while the file was growing, we saw the thin provisioning dropping. It dropped to +-60% and did not drop any further. We then deleted the zero.txt file and waited, to no avail. We reran the process of filling the area to 100% usage with a 'zero' file again and then deleted the file. We are still sitting at 60%.
What caused the thin provisioning to dynamically reclaim the capacity in the beginning ?
Why did the thin provisioning only drop to 60% when there is not data in the directory ?
Is there any additional steps we need to run to reclaim the rest of the capacity ?

[Richard Siemers]

I suggest running the test again with a single 560gb LUN to measure the difference. I suspect the reclamation process leaves some space allocated on a per lun basis for efficient re-growth needs. I would think it acceptable to keep some space allocated for this use, but 20g per lun seems too much.

I have seen similiar variances on windows LUNs post reclamation, however my LUNs are typically much larger so the % of space not reclaimed was much less. Next time I will record the actuall GB for comparison.

I am unaware of any minimun recomended lun sizes for thin reclamation purposes.

[Richard Siemers]

I've recreated a test at my site. What field are you using to measure "thin provisioning"? There are a couple, one is "Reserved User Size" and another is "Used User Size". From the CLI you will have to use the -s option on showvv to see both.

Your "Used User Size" should be mostly reclaimed as expected, but your RESERVED User size is probably still quite high? I'm looking into a way to unreserve/free up this space.

[rmurtagh]

I have observed the same effect on 1500GB JFS formatted LUNs.

These LUNs were 99% full. Most of the data was then deleted, leaving approx 5G. 'dd' was then used to create a large zero file. The "Used User Size" decreased as expected (down to 20GB) however the "Reserved User Size" initially decreased quickly, but then levelled out at about 45% (700GB). Using showvvmap and showldmap shows that this "extra" 700GB is still held by the VV and is not available for use by other VVs.

My feeling is that this is due to the mapping between the tpvv and the ld. According to the documentation the tpvv will reclaim "blocks" as small as 16kb filled with zero but suggests that you need 128kb of contigous zeros to release the space back to the tpvv. The problem comes because the VV is mapped to the LD in "regions" that are 64MB, 128MB or 256MB. (Most of mine are 128MB.) Therefore a single 16kb (or even 1 bit non-zero) of non-zero data can hold "reserved" a 128MB of LD space.

As a further test I then unmounted the filesystem and ran 'dd' directly to the raw LUN device. The "reserved user space" was quickly released, presumably as the filesystem overhead at the start of the LUN was zeroed out. Was was more interesting is that the regions shown in showvvmap were not released in strict order.

Towards the end of the RAW LUN wipe showvvmap displayed:

Code: Select all

Space    Start(MB) Length(MB) LdId       LdName LdOff(MB)
  usr  0       256        128  122 tp-2-sd-0.76    112640
       1    151680        128  127 tp-2-sd-0.78     48128
       2    151808        128  127 tp-2-sd-0.78     48256
       3    646400        128  130 tp-2-sd-0.79      5120
       4    647296        128  130 tp-2-sd-0.79      5504
       5    647680        128  130 tp-2-sd-0.79      5632
       6   1501568        128  137 tp-2-sd-0.88     12800
       7   1512320        128  137 tp-2-sd-0.88     22528
       8   1512448        128  136 tp-2-sd-0.85     28416

I suspect that there is another (invisible) mapping layer for tpvv's whereby 128kb blocks of the tpvv are mapped onto the LDs. This works well when the tpvv is growing by not allocating large amounts of LD for stray writes scattered over the disk, but seems to severly impede reclamation at the LD level. A (128MB) LD region is only released when there are no tpvv blocks allocated from it. There does not appear to be a "compact" function at the tpvv level.

[peckmanm]

Hey rmurtagh,

Any updates on this issue? We're seeing practically identical poor results attempting to reclaim space.

[NathanBell]

Richard you can correct me if I am wrong, but I thought the reclamation process was low priority and could occur over an extended period of time? At least that is what HP told us during training. The exception to this would be hosts that will issue SMI-S TRIM/UNMAP commands to the array (like windows 2012) and the chunklets are essentially just moved to available chunklets pool rather than reclamation having to kick in after the fact.

[Richard Siemers]

Depending on what you want to see, you need to watch different columns or data points... if you're doing thin reclamation and wanting to see your system free space grow, your going to have to reclaim a sizable amount of data, and wait for those background jobs to complete to see the impact. If you just want to see if your zeroing operation is working, then you should zero in on your VV's User Used space and watch it shrink in near real time.

When the background processes run to reclaim/defrag/etc the won't truncate a VV down to its absolute size... it leaves a portion of space reserved (aka User Reserved) for immediate growth/writes... I hypothesize that there is a threshold coded that says if that reserved space exceeds X% of the used size then un-reserve Y% and return it to the CPG. I believe a similiar task runs against a CPG with similar logic that returns excess reserved space to the system.