DPM 2012, Hyper-V 2012 and 3PAR Hardware VSS problems

[slink]

Anyone using this combination?

I have 4 Hyper-V hosts and 2 CSVs holding about 30 VMs between them. The CSV VVs sit on 1 RAID1 CPG and there is a separate virtual copy CPG which both CSV VVs are set to use for virtual copies (not sure if that is wrong but I didn't think it would be)

For me it's working but it's very intermittent and problematic. I get the following errors in the event logs on any of the hosts at different times:

3PARVSSProvider
3PARVSS5087: ERROR: Snapshot set is different than the previous one. Aborting prepare snapshot.

3PARVSS5054: ERROR: CLI: removevlun: VV vss0000004_rw_062013101857 not found

disk
The IO operation at logical block address 7ffff7ff for Disk 6 was retried.

The device, \Device\Harddisk6\DR28, is not ready for access yet.

volsnap
The flush and hold writes operation on volume timed out while waiting for file system cleanup.

VSS
A Shadow Copy LUN was not detected in the system and did not arrive.

LUN ID {5aafc9f1-8b00-4273-9c33-2a77b707e5ac}

Version 0x0000000000000001
Device Type 0x0000000000000000
Device TypeModifier 0x0000000000000000
Command Queueing 0x0000000000000001
Bus Type 0x0000000000000006
Vendor Id 3PARdata
Product Id VV
Product Revision 3121
Serial Number 60002AC0000000000000025E00004C1E

Storage Identifiers
Version 0
Identifier Count 0

Operation:
Exposing Disks
Locating shadow-copy LUNs
PostSnapshot Event
Executing Asynchronous Operation

Context:
Execution Context: Provider
Provider Name: HP 3PAR VSS Provider
Provider Version: 2.2.0.16
Provider ID: {5f466535-9c6c-4b33-be9f-8f3147381cf5}
Current State: DoSnapshotSet

I can run the protection job again in DPM and it will pass so seems like it's do with the 3PAR conflicting with itself. I thought the hardware VSS writer was supposed to be smart enough to work all this out. Using a software VSS there is talk of serialising the backups but no mention of needing to do that with a hardware VSS writer. I've tried reducing the number of parallel backups to 2 from 3 but makes no difference.

Anybody have a similar experience?

[hdtvguy]

What Inform OS are you at? I know there are a bunch of prereqs for Server 2012 and I think a new persona.

[slink]

This is brand new kit with the latest version and the host personas are all 2012.

[mnelson]

Did you ever find a fix for this? I have a very simular setup and I'm having the same errors.

[slink]

Nope. Disappointing as HP recently released a new VSS writer compatible with 2012 but the problems remain. Support said:

the VSS provider will only work with our RM products for windows 2012 backup products such as hp DP ,Netbackup, backup exec are supported with prior windows versions (i.e. windows 2008).

This means that for 2012 only our RMSQL and RM exchange 4.4 are designed to work. We do not know at this time if there will be a new version released to work with the backup softwares but one thing we did confirm is that Microsoft System center DPM is not supported on any of the platforms with our VSS provider.

I'm getting hold of a 90 day eval of Recovery Manager for Hyper-V to see if it's worth a purchase but I might just stick with scheduled virtual copy tasks on the 3PAR for guest backups and in-guest DPM agent backups for SQL, Exchange, files etc.

[hdtvguy]

IF RM-Hyper-V is anything like RM-VMware you will want to look for another solution. RM-vmware is a bit bloated and slow and if you have too many RC groups it will time out.

[MartinNr5]

Hi.

Perhaps I should start a new topic but I have the exact same question as the OP so I'll just bump this thread.

Does anyone know if there have been an update to this issue?

I'm coming from the DPM camp and have a customer that will change storage to 3PAR and I have to check if 3PAR has hardware VSS support before they go ahead and implement the new storage solution.

Thanks.

[slink]

There has been no change for me I'm afraid. The situation is if you use the hardware VSS writer you get the errors I describe and intermittent backup success. If you use the software VSS the cluster goes offline and all the VMs crash. So basically you can't backup using snapshots with DPM.

I tried HP's recovery manager for Hyper-V but as a previous poster suspected, it's crap, slow to the point of unusabliity.

So my backup of VMs is scheduled virtual copy tasks on each CSV VV and a manual restore if required. It's not so bad and only takes 10 minutes to get a server back. I've done it several times (see my other post about 3PAR corrupting my VMs with ODX switched on!)

[MartinNr5]

Thanks for the reply. We'll see what route we end up taking.

Perhaps you've seen this but here's a link to how you configure DPM to use software VSS: http://technet.microsoft.com/en-us/libr ... 34192.aspx

There's also a bug with the VSS provide that requires a hotfix on your Hyper-V hosts: http://support.microsoft.com/kb/2870270 (this one contain other useful hotfixes as well)

[slink]

DPM 2012 is not supposed to require serialization but I have read that some people report success with doing it but at the expense of a much larger backup window and also DPM taking ages to do consistency checks to find out where it's at. I don't like the potential risk or the extra management so native snapshots on the 3PAR is much more reliable for me.

We are about to use Hyper-V replica to another data center so that should remove any worries I have about losing the entire SAN. We've built in redundancy to Exchange (DAG) and SQL (AAG) so actually having to restore servers in the event of total SAN loss is unlikely anyway.

The hosts have all the available hotfixes and updates from MS. I spent a lot of time on trying to resolve this a couple of months back. The fixes made no difference and if you search you'll see several reports from people who have open cases with MS on this issue and the official word is that it's a problem and they're working on it, which was the outcome to my support case with MS. Hence why I gave up with DPM for snapshot backups on Hyper-V, it's just not reliable

Incidentally and OT, there are more problems with DPM that we've run into. Namely, you can't backup SharePoint 2013 if its databases are part of a SQL AAG (massive oversight - you can either have HA or recoverability but not both) and you can't back up a SQL AAG if one of the nodes is part of a failover cluster instance, so again, you can't have HA at the instance level combined with HA at the database level which gives you the flexibility you need for an HA+DR solution (FCI+AAG) as recommended in Microsoft's own white paper. Points to bear in mind if you're a consultant planning for solutions where you are using DPM.

I used to champion DPM as it was a great product but with the latest 2012 SP1 and 2013 product iterations from MS. it seems the product teams are not talking to eachother and we're ending up with crippled solutions if we want to embrace everything MS offers in each product.