HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 5 posts ] 
Author Message
 Post subject: expired vv snaps not auto deleting!
PostPosted: Mon Aug 17, 2020 4:00 pm 

Joined: Mon Mar 09, 2020 8:34 am
Posts: 67
Anyone come across this error before?

We have a scheduled task in SSMC which creates snaps every weekday for our live vv's. The command SSMC generates is:-

createsv -exp 168h -addtoset xx-vs-prd.Snapset @vvname@-@y@.@m@.@d@.@H@@M@ set:xx-vs-prd

Looks fine to me. However nothing gets autodeleted after the expiry date, the snaps just pile up and up and up. Its pathetic.

What I see is that the snaps which have not expired have a date of expiration, and when I run showvv -expired, there is a big list of vv snaps which have expired, however, it seems that despite running, the system task "remove_expired_vvs" (command: removevv -f -expired -matchbulkobjs) is not removing any of these at all.

I think we will never purchase another 3-par again, has anyone seen this behaviour before? We will have thousands of expired snaps. I see in SSMC the expired ones no longer have the expired bit enabled with a date beside it, but they show up as expired with the showvv -expired command, so really not sure what we do from here!
small example of expired:

xxxpar01 cli% showvv -expired
-Rsvd(MiB)- -(MiB)--
Id Name Prov Compr Dedup Type CopyOf BsId Rd -Detailed_State- Snp Usr VSize
423 xx-prd-3par-20.08.06.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
433 xx-prd-3par-20.08.07.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
443 xx-prd-3par-20.08.10.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
426 xx-prd-ad-20.08.06.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
436 xx-prd-ad-20.08.07.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
446 xx-prd-ad-20.08.10.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
424 xx-prd-hpe-20.08.06.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
434 xx-prd-hpe-20.08.07.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
444 xx-prd-hpe-20.08.10.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
427 xx-prd-sccm-20.08.06.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
437 xx-prd-sccm-20.08.07.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
447 xx-prd-sccm-20.08.10.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
425 xx-prd-vmware-20.08.06.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200
435 xx-prd-vmware-20.08.07.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200
445 xx-prd-vmware-20.08.10.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200


If this wasnt hundreds of thousands of £££££ of kit, I would really like to throw it out of the window and into a skip.

Ahh now I see the scheduled task has been failing for months, nice touch, its green and does not generate an alert, just PERFECT.

System task remove_expired_vvs, Task 10816, has failed.
04-Jul-2020 17:27:01 o'clock BST New
No user
Recommended Action
A system tasks has failed. Contact the HPE Support Center.
General
System wp3par01
Serial number CZ3815XKJ3
Type A system task failed
ID
Message code 0x0500001
Origin System

What a load of excrement.


Top
 Profile  
Reply with quote  
 Post subject: Re: expired vv snaps not auto deleting!
PostPosted: Mon Aug 17, 2020 7:46 pm 

Joined: Mon Mar 09, 2020 8:34 am
Posts: 67
So it appears that out of the box the standard system task of cleaning up expired snaps/vv's is pretty useless and does not work on snaps created from a vvset. Nice touch, super helpful.

The support guys said we should create a new schedule which runs every hour just before the standard system task, and use the -cascade option to remove all expired snaps from vv-sets.

Something like this:-

createsched “removevv -f -expired -cascade" "22 * * * *" removevv_expired_cascade

After checking the cli reference it says "
–cascade
Remove all the descendent volumes as long as none has an active VLUN. It will remove any VLUN
templates as long as there were no active VLUNs. It will remove the volumes from all the volume
sets. If the -expired option is specified, all expired volumes and their descendent volumes will
be removed regardless if they are expired or not."

Im a little dubious this might result in something unexpected, are we OK to go ahead with this custom schedule to remove expired snaps from our vv-sets?! :) Gulp! This is in prod now, and there cannot be any f-ups!


Top
 Profile  
Reply with quote  
 Post subject: Re: expired vv snaps not auto deleting!
PostPosted: Mon Aug 17, 2020 10:57 pm 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1328
Location: Dallas, Texas
T16 wrote:
createsched “removevv -f -expired -cascade" "22 * * * *" removevv_expired_cascade

After checking the cli reference it says "
–cascade
Remove all the descendent volumes as long as none has an active VLUN. It will remove any VLUN
templates as long as there were no active VLUNs. It will remove the volumes from all the volume
sets. If the -expired option is specified, all expired volumes and their descendent volumes will
be removed regardless if they are expired or not."

Im a little dubious this might result in something unexpected, are we OK to go ahead with this custom schedule to remove expired snaps from our vv-sets?! :) Gulp! This is in prod now, and there cannot be any f-ups!


Gulp! I feel your pain. You may want to test a small manual batch first. 'removevv -f -expired -cascade SomeDevServer_F_Drive*' and use the showvv command first to test a vv_name wildcard. You know the first time that schedule runs it's going to delete thousands of expired snaps at once. For piece of mind and control, you may want to clean that up in several smaller batches before setting up the new task.

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.


Top
 Profile  
Reply with quote  
 Post subject: Re: expired vv snaps not auto deleting!
PostPosted: Tue Aug 18, 2020 11:38 am 

Joined: Mon Mar 09, 2020 8:34 am
Posts: 67
phew it seems to have done the trick!

Skip now cancelled.. :) Hope this helps anyone in a similar position who uses vv-sets mapped to host-sets for ease of automation. The standard system task will not remove snaps in a vv-set, whereas the modified command shown above does the job if it runs ~5mins before the other standard system hourly expired vv task.

A cup of tea to celebrate :


Top
 Profile  
Reply with quote  
 Post subject: Re: expired vv snaps not auto deleting!
PostPosted: Thu Aug 20, 2020 10:45 am 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1328
Location: Dallas, Texas
Glad to hear all is well, and thank you for sharing!


My own personal opinion, and not necessarily the opinion of my employers, current or previous... I am not a FAN using vv sets in export templates because I prefer to have explcit LUN# assignment control. I ran into a situation where hosts needed both some provisioned LUNs for their own OS/swap/local use, and we also a member of a cluster that owned a VV:set. It worked fine, but some hosts had more dedicated LUNs than others... and it triggered our OCD when we saw:

Host1: LUN0-Boot LUN1-Scratch VVSET LUNS 2-10
Host2 LUN0-Boot VVSET LUNS 1-9

So Datastore1 was LUN2 on host1, but LUN1 on host2. Everything worked fine, it was just our own organic gray matter that took issue with it.

*disclaimer* this was been 8+ years on T800 arrays, very possible this has changed.

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 


Who is online

Users browsing this forum: No registered users and 38 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt