Possible to saturate the management port

[zappakat]

Morning everyone,

TBH I'm not really a 3PAR admin however I've hit a bit of an impasse with regards to monitoring the node and I'm trying to see what I can do to assist.

We have 4 3PAR 7400 modes and I'm trying to monitor them using the HP Storage Management Pack for OpsMgr.

The issue appears to be with the management port on the nodes. It can't cope with the traffic. The monitoring starts out with a SMI-S based discovery and is set it to run every 15 minutes which completes every time in just over 7 minutes. This consists of one powershell script. At the same time I can open the Inform console and connect to one of the arrays no problem.

However when I enable any of the monitoring (for example the cages monitor), this kicks off 20 powershell scripts per node (that's one script per cage) and is scheduled be default to run every 180 seconds. 80 powershell scripts every 3 minutes and that's just for the cages. Anyway this causes the port to saturate, the discovery starts timing out and I can no longer connect using Inform.

Even after disabling the monitor it takes at least 10 minutes for the port to start listening to requests again and the discovery to recover.

I've posted on the HP forum but there's been little reply.
Has anyone seen anything like this?

Thanks

[Richard Siemers]

I would drastically reduce your polliing intervals. I don't know the specifics of OpsManager or the scope of your project, but A full SMI-S discovery probably only needs to be done once a day, maybe even week... since the hardware config probably doesn't change very often.

Monitoring wise, can you throttle the number of either active connections and/or concurrent scripts to just 1 at a time?

[Cleanur]

Sorry I'm not familiar with Ops Manager but each controller node has a maximum session count per management port and only one management port is active at any one time. The number of concurrent sessions supported is relative per model for the 7000 I think it's 96 at present. So if you have lots of concurrent connections, especially if they don't terminate correctly then you will end up out of sessions. You can clear these with the "removeuserconn" command but it sounds like you need a method of staging the collection over a longer period of time.

[amorin]

We just started using the Storage Management Pack for SCOM. It basically rendered our 3PAR admin console useless. Even SSH connections were slow. I contacted HP support who found the hundreds of connections per second that were initiated from the SCOM server. As soon as I changed the SCOM user password everything was back to normal. The admin responsible for enterprise monitoring should be contacting support and I'll update if we hear anything. We are running V400 with 3.1.1 MU1.