7400c Very slow CLI on COM and SSH and dropping from MC/VSP

[senate014]

Hi there

I have a weird problem with a 7400c where after a couple of hours the systems CLI via SSH and Serial COM grinds to a very slow, sluggish performance when issuing any CLI commands as well as the system dropping it's connection intermittently from the Management Console and Virtual Service Processor.

Both Node's Mgmt RJ45 ports are plugged into a single Cisco Catalyst 4948 as well as the Hyper-V host server that holds the Management console and Virtual Service Processor VM's.

The Cisco Switch has a separate VLAN for the 3PAR Management interfaces that the Hyper-V host is also connected to.. So it's essentially isolated from the rest of the network via this VLAN.

The 7400c is running 3.3.1MU3 and is patched up to P116
It has 68 Virtual Volumes and a Snapshot policy that snaps all VV's every hour and retains the VV's for 4 days. Essentially each VV has 96 snapshots...

We have swapped out Node-0 already and plan to swap out Node-1 if we cannot figure out what the problem is here...

We have also tried running the "setsysmgr restart" command on several occasions which sometimes speeds up the CLI performance and stops it dropping from the MC/VSP...

One thing that did sort out the issue was stopping the Snapshot policies running.... but when we stared them backup again the CLI/SSH sessions started grinding to a holt along with the system dropping in and out of connection from the MC/VSP....

Rebooting one Node at a time also brings the performance back to normal, but again, it starts to slow and drop in and out from the MC/VSP after a couple of hours.

At no point do any host servers experience any issue with VV performance... It seems to be some sort of weird performance/connection issue with actually managing the system....

I've attached the current CPU usage....

Any suggestions or help would be much appreciated

Thanks

[MammaGutt]

It might be relevant, but weak CPU/low memory is why 7ks are recommended to stay at 3.2.2.

Did these suddenly appear or could they have been present for most of the time the system has been running 3.3.1?

Does all commands in CLI appear equally sluggish or are some worse than other?

[senate014]

It might be relevant, but weak CPU/low memory is why 7ks are recommended to stay at 3.2.2.

Did these suddenly appear or could they have been present for most of the time the system has been running 3.3.1?

Does all commands in CLI appear equally sluggish or are some worse than other?

Hi there and thanks for getting back to me.....

Most quick commands that display minimal information are okish... commands like "showtask" or "checkhealth" take forever...

I have just noticed that the 7400c disconnects from the MC rouhgly ever hour on the hour which is around the same time the hourly snapshots are taking place....

At the moment, we have also found that the system is unbalanced... the system had odd sets of disk in 4 out of 5 cages... 15x SAS Spinning Disk and 7x SSD's in 4 out of 5 cages.... The customer is going to install some more SSD and SAS Spinning Disk to rebalance the system by running "admithw" and "tunesys" once the drives have been installed.

I have also suspended all snapshot policies for the customer and the customer will be deleting all snapshots tonight so the system is back to the state where it just has the 68x base VV's.

The system has always been on 3.3.1MU3 P116.

Cheers!

[MammaGutt]

Just make sure when you rebalance that you get the drives in the correct order (they should be equal in pairs). So if there is SAS in the 15 first and SSD in the next 7 slots, you need to move the first SSD to get an even number of SAS drives in a row.

I find it hard to believe that the system has always been 3.3.1 MU3 P117 considering that was release maybe a year ago, while the final 7k left the HPE factory in 2016 .

But you are saying that this problem hasn't been there forever on 3.3.1 MU3 and that it happened at some point...

Does the SP disconnect happen at the same time as you snap? In that case I would be even more certain about resource issues.

Checkhealth and showtask are always slow as they do a lot of things. What about showpd, showport, showdate, shownet and statport?