Hi paulv666,
I've got some of the same equipment as you but connected in different ways. I've got an 8200 on 3.3.1 MU2 (originally 3.2.2 MU4) with deudpe enabled and doing iSCSI and my latency is ok given the number of hosts, volumes and vms but its not connected to the synergy.
I don't have a silver bullet for you but just wanted to toss out some ideas in the hopes that something will help you narrow down the problem:
1) If you have a bad cable, it could be causing high latency because of all the retransmits it might generate. I'm not sure how to check for retransmits on a 3par (my other san has a graph showing this value) but maybe use SSMC and make a realtime report for each port to see if one is obviously worse than the others?
Similarly, maybe run a statport at the 3par command line and see if one port has a higher service time than the others?
2) What interconnect are you using with Synergy? Virtual connect? Switch module? Do you have 2 frames connected together? Maybe traffic flowing through the interconnect is causing an issue if you have multiple frames? (You mentioned 6 hosts so I assume it's just 1 frame so maybe no interconnect). You don't have to answer with details. I know enough to be dangerous, not enough to be helpful
3) Trying disabling paths manually in ESXi so all traffic flows down a single path and then turn them back on 1 by 1 to see if latency goes up in case there's an issue with multipathing
4) You're not affected by this note on the synergy release set page are you? Users of HPE Synergy 40Gb F8 Switch Module are not recommended to use Synergy Custom SPP 2018.03.20180628. Contact your account team for further guidance.
5) Are you using a VSS or VDS? If VDS, are you sharing the same uplinks for iscsi and network? If so, are you using network i/o control?
6) Do you have iSCSI port binding set correctly if using a single subnet? If you have two subnets,it's recommended not to use port binding.
7) Disable delayed ack. I know there's nothing in the 3par docs that specifically say to do it but it's almost always a recommended thing to do as and as you discovered, doing it afterr the fact ain't fun
When I setup a new host I set my claim rule (iops=x and round robin) before doing anything with iscsi. Then add the IP addresses to the port binding section (I'm running with a single subnet) but don't do a rescan. I modify the delayed ack option on the ip in the dynamic section (could also be at the global level) and then reboot the host. If vmware takes a long time to get past the loading the iscsi stuff during boot I know there's a config mistake (doesn't necessarily mean there will be a performance problem but I know it means long rescan times because it's trying to access storage over a path that isn't valid)
8) If you're using jumbo frames, is it enabled end to end? Simple check I use is go on the ESXi host and ping the iscsi address of the 3par as follows:
Code:
vmkping -I vmk# -s 8972 -d <ip address of san port>
where vmk# is one of your iscsi ports.
9) Are you using software or hardware iscsi? I've only ever used the software iscsi adapter myself.
10) If you have support, you can always open a case and 3par support might have you run a perf script on your SP to gather data from the 3par. If you have phone home enabled, they will get the data and then can analyze it and check for duplicate packets/re-transmits.
11) Could your switch be dropping packets? Is storm controlled disabled?