Advise Required on SQl performace Impact with new 3Par P1000

[iredub123]

hi,
I'm looking for advise/guidance please.
we recently via a 3rd party vendor moved workloads/servers from local basic DAS storage (Dell MD1220s) via P2v and v2v migrations to a new 3par StoreServ 10400 hosted in a external datacenter. the solution has full DR capability with full synchronous replication to a 2nd 3Par storeserve installation. The production and DR data centres are approximately 24k apart and connected via dedicated multiple 1GB DWDM fibre links achieving sub 1ms latencies each way when measured at the fibre mux point to point level.

My question is this, for sql batch jobs that are (mainly rights) we are seeing roughly 30 to 40% increase since we moved onto the 3par in the time taken to process the jobs. This seems very high to me and I'm curious to understand if this would be the expected range of impact we would expect to see. I'm sceptical it is however Ive been informed that this is because of virtualisation (vm vsphere 5.1) and the 3par synchronization. It still seems very high to me.
any advice would be appreciated, I'm not a storage admin but I'm looking for a general direction if this is the norm or not and what we might do or consider to improve it.
thank you

[hdtvguy]

Is the P10400 a shared array I assume? Of it is your array just in a hosted DC? We would nee more info as to how the 3par is configured CPGs, is AO in play, etc.

[iredub123]

hi,
thanks for the quick reply, it's our array hosted in a DC and managed by vendor on our behalf, I'm not the storage admin so it's difficult to provide specs cuhs as CPGs et and AO (not sure what they are by the way as I'm not a admin), I'm more looking for high level guidance at this point if possible

[yizhar]

Hi.

Using sync replication might indeed be the cause, and you should first test if this is the bottleneck.

Keep in mind that sync replication requires every write operation to be completed on both copies, which in turn means that write operations to the synced volume are limited to 1gb bandwidth of the replication link = 100MB/sec.

So if your nightly jobs are write intensive, then sync replication and wan links are probably limiting factor.

If applicable, please try to pause/stop the replication or switch to async replication.

If you are performing SQL backup (full/log/whatever), I suggest that the target backup volume will not be part of sync replication.
Such configuration will probably provide much better performance.

Yizhar

[Richard Siemers]

Like Yizhar asked, the first question to find the answer to is if this is synchronous replication or periodic asynchronous.

The next questions to ask is what type if disk are the SQL servers in question sitting on, are they NL (Sata)? How are the CPGs configured for the ESX data stores? Are the SQL guests using VMDKs, RDMs/NPIV? Are the ESX data stores Thick or Thin, Eager or Lazy zeros? How many ESX hosts per data store, how many VM guests/VMDKs per data store?

The environment and problem you described is complex. Is your storage admin team separate or the same as your ESM/VM admin team?

[iredub123]

thanks guys for the information so far, very helpful.
some further information on CPGS for sql
***.SQL.FC.R1.01 Fibre 10 1+1 Cage Fastest SQL Production
***.SQL.FC.R5.01 Fibre 50 5+1 Cage Fast SQL Production (Default)
***.SQL.FC.R5.SNAP01 Fibre 50 5+1 Cage Snapshot space for SQL on Fibre drives
***.T2.SQL.NL.R1.01 NL 10 1+1 Cage High protection NL SQL
***.T2.SQL.NL.R6.01 NL 60 6+2 Cage Good protection NL SQL (Default)
***.T2.SQL.NL.R6.SNAP01 NL 60 6+2 Cage Good protection NL SQL Snapshot
I believe the disk that the production sql is sitting on is Eager-zero Thick. AO is employed but I don't have the detail on the various polices. The storage team is separate. I've been told the virualisation technology in vmware is adding the large percentage performance impact in latency. find that hard to believe. appreciate the help folks and I understand you need more detail that I can't provide, again just seeking broad advice.

[Josh26]

I see three FC CPGs there and it's entirely possible someone thinking they were smart restricted each of them to dedicated disks, which can only ever hurt performance.

As far as virtualization - no. It can however be configured incorrectly. Can you see what SATP and PSP are in use?

There was a recent thread on this forum suggesting for certain HBAs, the very latest drivers were needed in order to assist with performance issues.

[iredub123]

is there any tool you guys could remember that can break down a service request and show the various latency elements ? for example it would be very useful to be able to pin point the amount of latency introduced through the remote copy to the DR site.
Also can you tell me if the 3par can be setup to do both synchronous and asynchronous remote copies for different work loads ?

[hdtvguy]

3par currently only support synchronous or periodic. Periodic requires that you set intervals to replicate and has numerous short comings if you read my other posts. If you can work around those limitations then periodic is the way to go. System Reporter is a great tool, but does not have any replication related reports that would help you narrow down the latency. Unless your data need 100% guarantee of being synchronous you should use periodic.

[iredub123]

that's great, thanks for all the help