Using the adaptive queue depth algorithm in EXS

[sremsing]

Is any one using VMware's adaptive queue depth algorithm (AQDA) with 3PAR storage? I am setting up a new environment and looking for guidance.

Today we have eight vSphere 4 (ESXi) servers connected to a two node F400. At this time we do not plan to connect any other servers to the F400 nor do we plan to connect any other storage to the ESX servers. All the ESX servers have Emulex LPe11002 HBAs.

Given that design would you use AQDA? If so, what would you set the Disk.QFullSampleSize and Disk.QFullThreshold value to? If you would not use AQDA what would set the HBA's target throttle value to?

In either case what would you set the Disk.SchedNumReqOutstanding value to?

Also, is there any change I need to make on the HBA (i.e. firmware settings) when changing the ESX queue depth.

Thanks

[Richard Siemers]

Hello SRemsing,

The safe answer is "It Depends".

Adaptive queue throttling can occur from either PORT congestion OR LUN congestion. So depending on how your environment uses the storage, and how busy it is will determine if you need it or not.

When the number of QUEUE FULL or BUSY conditions reaches the QFullSampleSize value, the LUN queue depth reduces to half of the original value. When the number of good status conditions received reaches the QFullThreshold value, the LUN queue depth increases one at a time.

For 3Par arrays its recommended that QFullSampleSize is set to 32, I would suggest QFull
Threshold to be set at 8. No HBA settings required that I know of.

Disk.SchedNumReqOutstanding is a per virtual machine setting (CORRECTION: its a system wide setting that effects Queue Depth per virtual machine --RS ) .

If you havn't already, take a look at: http://kb.vmware.com/kb/1008113

[sremsing]

Thank you for the quick reply.

Your explanation of the QFullSampleSize conflicts with others that I have seen or been given. Some have suggested that the QFullSampleSize is the per LUN queue depth. I have not been able to find anything from VMWare that defines these parameters. Do you have a reference? Assuming the QFullSampleSize is as you described (and I believe it is), what is the per LUN queue depth that the server will use and how do I set it?

Also, Disk.SchedNumReqOutstanding is not a per VM setting, it is set on the ESX host. Documentation from either 3PAR or VMware (I made a note of it but can't find the original source) suggested setting it to the same as the queue depth. If I understand this parameter correctly that could be a bad decision if you have a single VM spike in I/O traffic. Couldn't the other VMs on the same ESX host face I/O resource starvation?

[sremsing]

I just reread http://kb.vmware.com/kb/1008113 and found it contains the following description of Disk.QFullSampleSize and Disk.QFullThreshold

When enabled, it works by examining the completed I/O operations that fit within a window of Disk.QFullSampleSize operations. If the count of queue full errors is equal to or more than the value of Disk.QFullThreshold within that window, then the queue depth is throttled.

Also, I found the reference to the Disk.SchedNumReqOutstanding pararmeter. It is on page 3 of VMware's Scalable Storage Performance document. This was written for ESX 3.5 and not has not been updated for 4.0 (yet).

[Richard Siemers]

Some have suggested that the QFullSampleSize is the per LUN queue depth. I have not been able to find anything from VMWare that defines these parameters. Do you have a reference?

http://kb.vmware.com/kb/1010922 and http://kb.vmware.com/kb/1267 might be the references you are looking for. QFullSampleSize is not the physical HBA queue depth its a sampling window, queue depth is set from the ESX console (see http://www-dl.emulex.com/support/vmware/734/vmware.pdf ) and is the target parameter being throttled by the algorithm.

I did misunderstand Disk.SchedNumReqOutstanding, you are correct it is a system wide setting. However it sets the per VM lun queue depth globally (not individually like I assumed). I agree with you, I would prefer to have this set to a fraction of the physical HBAs queue depth based on the ratio of HBA ports to VMs. The documented defaults may assume you have multiple active fibre ports. If you have 4 active HBAs then thats a total of 4 times the physical queue depth. Having the Disk.SchedNumReqOutstanding setting equal to the Emulex disk queue setting would result in 1 VM being limited to 25% of the available hardware.

[sremsing]

Richard,

Thank you for your replies. We are going to test the adaptive queue depth algorithm with the following values:

Emulex HBA driver lpfc_lun_queue_depth = 32, 64, 128 (we are going to test each)
Disk.QFullSampleSize = 32
Disk.QFullThreshold = 8
Disk.SchedNumReqOutstanding = half the per LUN queue depth on the Emulex HBA