HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 15 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Slow VM Restores to Thin 3PAR VVs
PostPosted: Tue Apr 28, 2015 8:40 pm 
User avatar

Joined: Fri Sep 27, 2013 9:08 pm
Posts: 5
Hi,

We have a HP 3PAR 7400 4-Node with 128 x 600GB 10K SAS Disks connected via 8Gb Fibre to two HP SN3000B Fibre Switches. We have two 16TB Thin Provisioned VVs presented to a 7-Node ESXi 5.5 Cluster with about 80 VMs.

We use Veeam Backup & Replication 8 Update 1 and backup all VMs using the Storage Snapshot Integration which works well. The Veeam Proxies and ESXi Hosts are HP ProLiant BL460c Gen8 Blade Servers with FlexFabric 10Gb LOMs (configured only for 10Gb Ethernet) and a Dual Port 8Gb Fibre Channel HBA which connects to the Fibre Switches.

We backup to a HP StoreOnce 4430 NAS Share with Veeam. A single backup can run at upwards of 400MB/sec as we have 10GbE between all server infrastructure.

However, when restoring we see terrible speeds, around 11MB/sec. We did some testing and found some interesting results. If we restore a VM using Veeam to a 3PAR Volume that is thick provisioned, we see upwards of 130-140MB/sec restores.

If we restore to one of our existing 16TB Thin Provisioned 3PAR Volumes we see a restore speed that never goes above 11MB/sec. It's almost like it's capped at 100Mb.

We did some further testing and presented a new VV from the 3PAR as thin provisioned. We restored to it and saw 130-140MB/sec restore speeds. Once the restore was finished we then did another restore and the speed again dropped down to 11MB/sec.

We converted the VV to Thick Provisioned on the 3PAR and did another restore and the speed went back up to 130-140MB/sec.

We repeated this a number of times with new test VVs from the 3PAR and found the same results each time. The first restore to a thin VV is fast, but all subsequent restores drop to 11MB/sec. If the VV is thick provisioned the restores are consistently fast.

I understand with a thin provisioned VV that there is some overhead as the 3PAR allocates the chunklets on the fly as the VV is expanding, but would not expect this much of a difference. If I storage vMotion a Virtual Machine between our two 16TB Thin VVs it is very fast. Only a restore seems to be affected.

There is a post on the Veeam Forums from 2013 where a couple of people saw the exact same issue: http://forums.veeam.com/veeam-backup-re ... 87-15.html. I have posted a message on that topic as well as there was no answer given on that thread.

For reference our 3PAR is running OS 3.1.2 (MU3) and has patches P18, P25, P33, P35 and P39 installed.

I've tried multiple Veeam proxies, both physical and virtual, and always see the same results. This one has me baffled at the moment.

Has anyone ever seen something like this?

Thanks,

Chris


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Wed Apr 29, 2015 3:50 pm 

Joined: Wed Nov 19, 2014 5:14 am
Posts: 505
It's a guess but there was similar issue in earlier versions of firmware related to how Windows 2012 handles a quick format. Basically it sent unmap commands for every block on disk just in case the volume had been previously written to. The firmware now recognizes this and sends a completion code since the blocks are already zero'd out, so no need assign the block and then unmap.

The fix is in the Windows Integration User Guide page 63
http://h20628.www2.hp.com/km-ext/kmcsdi ... 621-14.pdf

#fsutil behavior query DisableDeleteNotify 1
1 – Disable UNMAP
0 – Enable UNMAP(default)

Appreciate you're running ESX but I'm unsure where you're actually restoring to, it could be the previous image deletion prior to restore is forcing unmaps or similar before new data can be written to disk forcing the restore to queue.

Alternatively despite the delete, the file system may not actually overwrite the old data but instead allocate net new blocks meaning you'll incur a first write penalty as the new blocks are allocated. Maybe try restoring to a smaller volume where there isn't the same scope to not overwrite blocks. You could also try unmapping the blocks in the datastore prior to the restore and see if that has an effect.

I've seen similar issues on other thin provisioned systems with inhouse batch scripts that create a lot of churn on disk e.g create/merge/delete/create etc most were typically related to how the file system handled metadata updates and their interaction with thin volumes rather than the array itself.


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Wed Apr 29, 2015 9:10 pm 
User avatar

Joined: Fri Sep 27, 2013 9:08 pm
Posts: 5
Thanks for the reply John. I think you are on the money.

I did some more reading and tests and found the following:

- Restored to a test Thin VV and the restore was fast, then deleted the VM
- Restored again and saw the restore go very slow
- Ran the "esxcli storage vmfs unmap" command on the Datastore
- Restored again and the restore was still slow
- Found we had the "VMFS3/EnableBlockDelete" set to 0, when it should be set to 1, so changed it to 1
- Re-ran the unmap command and verified using ESXTOP that delete operations were sent to the storage
- Restored again and the restore was fast

To further confirm the issue is related to writing over the top of blocks that the 3PAR believed had data in them, I did some further tests:

- Deleted all test restore VMs
- Ran the unmap command on ESXi
- Restored a VM and it was fast, but cancelled the restore after 7GB
- Deleted the restored VM but did NOT run the unmap command
- Restored the VM again and for the first 7GB it was very slow again, but then picked up after that ran at the fast speed

This leads me to believe that what you are saying sounds very similar to what is happening, with ESXi believing the blocks are free and trying to write to them, but the 3PAR going "hey there is data here" and this causing the delays. Once the restore gets to blocks that the 3PAR hasn't written to yet then the restore returns to the expected fast speeds.

So, with all of that the question now comes to what is the solution. We have noticed in Veeam that even though a VM that is backed up is THICK EAGER ZEROED (which is recommended for thin storage), when you restore the VM in Veeam even though you tell it to keep the disks "Same as source", the disks are restored as THICK LAZY ZEROED. EAGER ZEROED actually zeroes out every block before you use the VM so there is no penalty on first write. This adds overhead to the VM creation time, but gains you performance when the VM is being written to.

Perhaps if Veeam would restore VMs as THICK LAZY ZEROED we wouldn't see this issue.

Since ESXi 5.0 Update 1 VMware disabled automatic unmap commands being sent to the storage array as it caused problems at the time. The only way to issue an unmap is manually (or via a scheduled powershell script). Perhaps this is what we should run, say once a week, across all of our 3PAR datastores.

Chris


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Thu Apr 30, 2015 3:53 am 

Joined: Wed Nov 19, 2014 5:14 am
Posts: 505
This is "sort of" expected behavior for any TPVV on any system but it may be a weird interaction. So I'd go back to Veeam and ask them to work with HP. It may be there's a simple fix, unmap seems to be a workaround but it might be a corner case that needs something in firmware to provide a permanent fix.


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Thu Apr 30, 2015 6:27 am 

Joined: Thu Dec 06, 2012 1:25 pm
Posts: 138
Sounds to me like an issue with with the ld row-size, a sequential restore will do a single stream read/modify/write to the underlyingLDs, which if the row-size is too low, can be not very good for performance (you are only touching one ld at the time, so you are limited to the performance of the drives the ld is striped too).
A fresh write will be better balanced across all nodes, and also not need the r/m/w-cycle, thus giving a better performance.

As you are still on 3.1.2MU3, you could well have issues with the growth size.
Could you post (or pm) for me the output of:

showcpg -sdg
showcpg -sag

showvvmap <name-of-the-vv> (dirty, before unmap please)
and finally a showld -d <ld-name> of the LDnames shown by the showvvmap command.

With that info i can see if my assumption is correct or not.

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Thu Apr 30, 2015 6:12 pm 
User avatar

Joined: Fri Sep 27, 2013 9:08 pm
Posts: 5
Thanks for the replies. I've updated Veeam on the results of the tests and are waiting to see what they come back with. It's in their interest for 3PAR Restores to be efficient since they push the fact they integrate with 3PAR pretty hard.

I've run the commands. I've attached the results of each of the outputs.

The showvvmap for one of the VMware Datastore VVs (16TB Thin VV) returned over 77,000 rows so it's not feasible to run showld on all of them. I picked the first 4 and ran it on them tho.

Let me know if this is what you were after.

Thanks,

Chris


Attachments:
002 - ShowVVMap.zip [737.6 KiB]
Downloaded 2639 times
003 - ShowLD.txt [2 KiB]
Downloaded 2901 times
001 - ShowCPG.txt [622 Bytes]
Downloaded 2976 times
Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Fri May 01, 2015 4:16 am 

Joined: Thu Dec 06, 2012 1:25 pm
Posts: 138
Ok, growth sizes look ok, with their default values. I might actually have been wrong with my assumption.
Totally right, do not post all 77000 lines, that is bonkers, didn't think of the volume being that large, sorry :)

The showld commands are actually from the sa-lds, which house the volumes metadata.
sorry, i should have clarified i was looking for a few of the SD-lds, as the SA-LDs do not give me any interesting values.

Could you give me the output of showld -d of just these:
tp-3-sd-0.13
tp-3-sd-0.14
tp-3-sd-0.15
tp-3-sd-0.16
tp-3-sd-0.17
tp-3-sd-0.18
tp-3-sd-0.19
tp-3-sd-0.20

As these LDs house the majority (>90%) of your data. if these show a row-size of 5 or 6 (and not 1), i'm definitively on the wrong track, and it is something else.

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Mon May 04, 2015 7:32 pm 
User avatar

Joined: Fri Sep 27, 2013 9:08 pm
Posts: 5
Here is the showld of the disks you listed. Thanks for your help with this.

I'm not sure if I'm happy or not that your assumption may not be accurate ... I kind of wish there was a known solution here.

So far the only thing we can do is just run the vmfs unmap command before attempting any restore as things like Storage vMotion, VM Snapshots, deleting a VM, etc all contribute to causing VM restore issues with TPVVs.


Attachments:
004 - ShowLD.txt [4.05 KiB]
Downloaded 2875 times
Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Tue May 05, 2015 1:46 am 

Joined: Thu Dec 06, 2012 1:25 pm
Posts: 138
Row size is 8, so these LDs are actually striped across all FC drives behind the node (32). I had the wrong hunch, sorry.

Without really detailed performance logs from both a clean write and a restore to existing space, i'm unable to pinpoint the issue i'm afraid :( .

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.


Top
 Profile  
Reply with quote  
 Post subject: Re: Slow VM Restores to Thin 3PAR VVs
PostPosted: Tue Mar 22, 2016 10:43 am 

Joined: Tue Mar 22, 2016 10:39 am
Posts: 2
Very interested to know if you ever got to the bottom of the slow restores to thin provisioned virtual volumes. Have seen much the same thing with a 7200c and vSphere 6.

Thanks

Adam


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 15 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: No registered users and 59 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt