I have two 7400 4 node 3PARs, one PROD, one DR. The RC-IP GigE port on each node is connected through a switch to a RC-IP port on the DR nodes (4 in use on each 3PAR). A CISCO switch takes in the 4 drops from each 3PAR at each location. There are 2 1GbE fibers etherchanneled between the two switches. Data is flowing equally across each of the two fibers etherchanneled together.
Short distance between the two arrays currently as I sync them up. Our CCIE is confident that CISCO gear is configured and working properly. I am only seeing about 900Mbits/s total spread across etherchannel. If there was only one RC-IP port connected on each 3PAR I would have expected the flow to stay on just one side of the etherchannel, but I have 4, each with a different IP so the flows should be shared across the etherchannel (and are).
I am going to need to get way more than 900Mb/s between these two arrays if I am going to keep these two reasonably in sync. What have you all seen in the way of throughput over your IP links when using RC-IP for remote copy?
What sort of throughput have you seen with RC-IP?
Re: What sort of throughput have you seen with RC-IP?
Please note that you will not see double the performance by using double the cabling in LACP through etherchannel. Your Cisco guy seeing 900/mb should be about right.
-
spencer.ryan
- Posts: 35
- Joined: Tue Feb 11, 2014 11:33 am
Re: What sort of throughput have you seen with RC-IP?
What's the hashtype set to for the etherchannel?
If there are multiple streams it should use both physical links. I'm not sure if that's how a 3par does it however.
If there are multiple streams it should use both physical links. I'm not sure if that's how a 3par does it however.
Re: What sort of throughput have you seen with RC-IP?
It is using both physical links, fairly evenly. CCIE tells me that as long as the flows have different source/destination IPs that should be able to travel over both links which is what we are seeing. I will check the things you suggested.
Has anyone ever used the 3PAR CLI checkrclink command to test throughput?
I reviewed this thread at CISCO, seems to makes sense:
https://supportforums.cisco.com/thread/2075664
statrcopy output:
14:12:18 02/25/2014 -Throughput(KBytes per sec)-
Target Node Address IPC Total(KBytes) Current Average
-----------------------------------------------------------------------------------
3par-mgmt-dr 0:3:1 10.xxx.xx.124 RCs05 6660091748.23 27269.32 28071.64
3par-mgmt-dr 1:3:1 10.xxx.xx.123 RCs17 6449301735.18 28220.37 29709.13
3par-mgmt-dr 2:3:1 10.xxx.xx.122 RCs21 6720985897.68 29171.32 29331.11
3par-mgmt-dr 3:3:1 10.xxx.xx.121 RCs32 6446473510.25 28927.80 29798.13
-----------------------------------------------------------------------------------
3par-mgmt-dr 26276852891.34 113588.81 116910.02
receive 0:3:1 receive RCr06 7534946.52 11.04 10.93
receive 1:3:1 receive RCr18 7233860.71 12.65 12.11
receive 2:3:1 receive RCr23 7380551.63 11.61 11.31
receive 3:3:1 receive RCr34 7253750.52 11.25 11.06
-----------------------------------------------------------------------------------
Receive 29403109.38 46.55 45.42
Send 26276852891.34 113588.81 116910.02
-----------------------------------------------------------------------------------
Total 26306256000.72 113635.36 116955.43
Has anyone ever used the 3PAR CLI checkrclink command to test throughput?
I reviewed this thread at CISCO, seems to makes sense:
https://supportforums.cisco.com/thread/2075664
statrcopy output:
14:12:18 02/25/2014 -Throughput(KBytes per sec)-
Target Node Address IPC Total(KBytes) Current Average
-----------------------------------------------------------------------------------
3par-mgmt-dr 0:3:1 10.xxx.xx.124 RCs05 6660091748.23 27269.32 28071.64
3par-mgmt-dr 1:3:1 10.xxx.xx.123 RCs17 6449301735.18 28220.37 29709.13
3par-mgmt-dr 2:3:1 10.xxx.xx.122 RCs21 6720985897.68 29171.32 29331.11
3par-mgmt-dr 3:3:1 10.xxx.xx.121 RCs32 6446473510.25 28927.80 29798.13
-----------------------------------------------------------------------------------
3par-mgmt-dr 26276852891.34 113588.81 116910.02
receive 0:3:1 receive RCr06 7534946.52 11.04 10.93
receive 1:3:1 receive RCr18 7233860.71 12.65 12.11
receive 2:3:1 receive RCr23 7380551.63 11.61 11.31
receive 3:3:1 receive RCr34 7253750.52 11.25 11.06
-----------------------------------------------------------------------------------
Receive 29403109.38 46.55 45.42
Send 26276852891.34 113588.81 116910.02
-----------------------------------------------------------------------------------
Total 26306256000.72 113635.36 116955.43
Re: What sort of throughput have you seen with RC-IP?
Aggregating x-times does not give you x-times the speed!! So please keep in mind that 4x10G is not 40G, but 4x10G. You need multiple flows and depending an the load balancing that won't even work.
So basically each packet is hashed and the hash results will be assigned to an egress port. Hashing occurs in your server on sending and again in the switch. Now both may use different fields to hash, so even if you hash well on the NIC you may hash to the same link on the switch. Also keep i mind that there are four links, so the chance of hashing two streams on the same link is 25% for each hop!
Use "show port-channel load-balance" (<--Cisco specific) to see what the switch does.
The reason why we hash in the first place is to ensure packet order on different link lengths in a channel. I think you can force Linux to round-robin style, but the switch will serialize again. Keep in mind that out-of-order packets will either blow the receiving network stack or completely kill performance.
Your only options are to use multiple flows (also look into TCP-Multipathing, new kernel feature) or to use faster interfaces.
Now, I know the 3PAR runs Linux, and I'm sure this isn't able to be done without root to the nodes. This may be a question for 3PAR Tier 3 support or so.
So basically each packet is hashed and the hash results will be assigned to an egress port. Hashing occurs in your server on sending and again in the switch. Now both may use different fields to hash, so even if you hash well on the NIC you may hash to the same link on the switch. Also keep i mind that there are four links, so the chance of hashing two streams on the same link is 25% for each hop!
Use "show port-channel load-balance" (<--Cisco specific) to see what the switch does.
The reason why we hash in the first place is to ensure packet order on different link lengths in a channel. I think you can force Linux to round-robin style, but the switch will serialize again. Keep in mind that out-of-order packets will either blow the receiving network stack or completely kill performance.
Your only options are to use multiple flows (also look into TCP-Multipathing, new kernel feature) or to use faster interfaces.
Now, I know the 3PAR runs Linux, and I'm sure this isn't able to be done without root to the nodes. This may be a question for 3PAR Tier 3 support or so.
Re: What sort of throughput have you seen with RC-IP?
SW-SOC1-DC-CORE-01-sp#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
EtherChannel Load-Balancing Configuration:
src-dst-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
Re: What sort of throughput have you seen with RC-IP?
I think we do have multiple flows and I am told that both physical links are being utilized fairly evenly.
Re: What sort of throughput have you seen with RC-IP?
checkrclink shows that all RC-IP ports individually can push 900,000Kb +/- 500. But, when tested together they are evenly reduced to a sum of 900,000. I plan to procure a couple of 10Gb switches to eliminate the ether-channel from the equation.
Re: What sort of throughput have you seen with RC-IP?
We recently asked 3par for some real limit numbers. Their answer of course depends on your connection speed and latency. The number they tend to say they will say is a supportable number is 125MB/sec for a V400 with potential bursts up to 175MB/sec.
if you run statrcopy -hb you can see the throughput currently and average. We can consistently get 70-90MB/sec, but I will tell you that is when 20 volumes are going concurrently as the array is nto very efficient with a handful at a time.
Our WAN is 300 mb with 24ms latency and we use Cisco accelerators that give us a 3x - 5x increase on average in throughput.
if you run statrcopy -hb you can see the throughput currently and average. We can consistently get 70-90MB/sec, but I will tell you that is when 20 volumes are going concurrently as the array is nto very efficient with a handful at a time.
Our WAN is 300 mb with 24ms latency and we use Cisco accelerators that give us a 3x - 5x increase on average in throughput.
Re: What sort of throughput have you seen with RC-IP?
Thanks for the info. My network team found a configuration issue and changed things to insure that things flowed properly - I think they basically assigned two source IPs to one link and two to the other, but I am not positive. As soon as I find out what exactly that was I will share the fix. At any rate, I am now, if I am doing the math right, I am getting about 1.6Gb/s of throughput, almost twice as much as I was.
r/w I/O per second KBytes per sec
Port Cur Avg Max Cur Avg Max Errs Drops
0:3:1 t 35546 29172 56776 32493 26913 51795 0.0 0.0
1:3:1 t 52650 37398 56814 48939 35040 53495 0.0 0.0
2:3:1 t 82597 53841 86540 80624 53024 84329 0.0 0.0
3:3:1 t 52201 38366 58445 48613 35946 54881 0.0 0.0
------------------------------------------------------------
total t 222994 158777 210669 150922 0.0 0.0
Press the enter key to stop...
r/w I/O per second KBytes per sec
Port Cur Avg Max Cur Avg Max Errs Drops
0:3:1 t 35546 29172 56776 32493 26913 51795 0.0 0.0
1:3:1 t 52650 37398 56814 48939 35040 53495 0.0 0.0
2:3:1 t 82597 53841 86540 80624 53024 84329 0.0 0.0
3:3:1 t 52201 38366 58445 48613 35946 54881 0.0 0.0
------------------------------------------------------------
total t 222994 158777 210669 150922 0.0 0.0
Press the enter key to stop...