3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: Remote Copy snapshots
PostPosted: Thu Oct 17, 2019 7:46 am 

Joined: Thu Oct 17, 2019 6:22 am
Posts: 4
Hello 3par guys,
excuse me for maybe just stupid questions. But I would be very grateful for a little clarification of some things.
I have two 8200 systems with async periodic sync configured. So far for now I have configured three remote copy groups, at this moment each group contains single VV. initial sync ran well and now we are syncing in 10min interval.
The thing I just messing my head with is remote copy snapshot size.
Not sure if I understand well to how snapshot mechanism with remote copy works. Yes I read the remote copy guide ;)
On source side there is a resync snapshot maintained for each synced VV. This snapshot reflects the state before the last sync interval. When next sync interval starts, another snapshot is created on source side and new one snap is created on target side.
When sync is finished. Target snapshot is deleted and on the source side older of the two copies is deleted.
From what I understand, I thought the size of source snapshots is equal to change rate of replicated production VVs. But when I check for showvv -s.
Snap space uses much more then I expect, what is confusing to me, that even after succesfull sync the used space practically not changing.
For example - VV with 8192GiB size has 1592Gib rsvd snap space, and after sync its still the same.
And yes Iam 100% sure that I dont have 1,5TiB change rate at this VV.
There are no other snapshots on this VV.
Thanks in advance to anyone for shedding some light to my "problem".
Regards
Martin


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Thu Oct 17, 2019 2:14 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 969
Location: Europe
Space is reserved as needed and shrink thru normal reclaim process.

Which 3PAR OS version are you running and what is the output of showvv -s of the volume?

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Fri Oct 18, 2019 2:10 am 

Joined: Thu Oct 17, 2019 6:22 am
Posts: 4
System is currently running 3.3.1 MU2.
You say the space is reserved and then used as needed. So when I run initial sync and snapshot allocates 2TB of space, it remains allocated even when next sync uses new resync snapshots with few gigs of real space used?
I attached showvv -s

Attachment:
showvv-s.PNG
showvv-s.PNG [ 22.38 KiB | Viewed 803 times ]


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Fri Oct 18, 2019 10:21 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 969
Location: Europe
Hi.

Run the following commands in CLI:

showeventlog -debug -oneline -msg "Reclaim History" -min 1440
showeventlog -debug -oneline -msg "cand" -min 1440

The first one will show successful tasks reclaiming space for all volumes the last 24 hours.
The second one will show output from defrag which runs every time the reclaim task in unsuccessful in reclaiming any data.

The reclaim process should run multiple times per day for all volumes with more than 4 GiB of data per node to reclaim. As you have a 8200, any volumes with more than 8192 MiB in difference between Usr Rsvd and Usr Used should reclaim which the two first volumes fit.

My assumption is that your system is not running reclaim or defrag on any of your volumes. 3.3.1 MU2 P76 contains a fix for this (which is superseeded by P96 and P103), so if you are below 3.3.1 MU2 P76, install 3.3.1 MU2 P103 and run the following commands in CLI:

freespace -f CA-NL-01
freespace -f FS-NL-02
freespace -f VM-NL-01
freespace -f VM-SSD-01

If you are above 3.3.1 MU2 P76, just issue the freespace commands.

Then do another showvv -s and see if the "Snp Used" column has changed.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Mon Oct 21, 2019 7:46 am 

Joined: Thu Oct 17, 2019 6:22 am
Posts: 4
Hi MammaGutt,
I checked for tasks and it shows that no reclaim or defrag events ran in last 24 hours.
My system is running P32,P40,P45,P51,P52,P60,P76,P78,P90,P96.
P103 is not installed so far. As you wrote - It seems the system is not runnin any reclaim task.
I havent noticed this kind of issue was addresed in P96 or 76 :(
I am considering running freespace command, but, since I use remote copy, I think it wont work with existing child snapshot.
Anyway, thanks for your time and help.
Regards Martin


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Mon Oct 21, 2019 9:50 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 969
Location: Europe
MartinV wrote:
Hi MammaGutt,
.........
I am considering running freespace command, but, since I use remote copy, I think it wont work with existing child snapshot.
Anyway, thanks for your time and help.
Regards Martin

That was my comment as well, until I ran the command and the snapshot stats changed and reclaim started running again.

I would however install P103 first as P96 was pulled and replaced by P103.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Tue Oct 22, 2019 8:01 pm 

Joined: Tue Jan 22, 2019 2:06 pm
Posts: 26
Location: Calgary, AB
One of the recent OS updates has added some serious space reclamation optimizations. We had issues of reserved space not freeing up at all, now that we have the latest patches (as of about a month ago) We have claimed back about 10TB of storage that was in "reserved" space.


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Fri Oct 25, 2019 3:11 am 

Joined: Thu Oct 17, 2019 6:22 am
Posts: 4
Hi MBILC,
thanks for info.
Just for update. I am already in touch with HPE about latest patch to our 331MU2 (p103).
As I checked for release notes of few latest patches i wasnt able to find related issue.
Do you have any more detailed information about exact patch? Thanks


Top
 Profile  
Reply with quote  
 Post subject: Re: Remote Copy snapshots
PostPosted: Fri Oct 25, 2019 12:24 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 969
Location: Europe
MartinV wrote:
Hi MBILC,
thanks for info.
Just for update. I am already in touch with HPE about latest patch to our 331MU2 (p103).
As I checked for release notes of few latest patches i wasnt able to find related issue.
Do you have any more detailed information about exact patch? Thanks


3.3.1 MU2 P76
Quote:
Issue ID: 209210, 263132
Issue summary: Used space for snapshots is calculated incorrectly.
Platforms affected: All StoreServ
Affected software versions: 3.3.1 GA - MU2
Issue description: The used space for snapshots is misreported due to the number of free pages being
incorrectly calculated. This is evident from the showvv -s output.
Symptoms: Incorrect statistics in showvv -s.
Conditions of occurrence: Writes and zero writes to volumes with snapshots.
Impact: Low
Customer circumvention: None.
Customer recovery steps: Run offline checkvv to fix existing bad statistics.


What I was told is that freespace will fix most of those issues without having to take the volume offline.

As for my recommendation on P103

Quote:
Issue ID: 281444, C23048
Issue summary: Executing freespace or setvv to remove the snap_cpg space on compressed, decompressed or
TDVV volumes results in multiple controller node down and array down and data unavailable situations.
Platforms a‚ected: All StoreServ
A‚ected software versions: 3.3.1 MU2
Issue description: Unhandled conditions in compression or dedup VVs result in controller node down conditions when
setvv is executed on these volumes to remove the snap CPG. Sometimes multiple controller nodes go down
simultaneously leading to array down and data unavailable conditions.
Symptoms: Node and cluster down condition with the above mentioned patches with stack trace indicating compression
code paths.
Conditions of occurrence: setvv on compressed, decompressed or TDVV volumes with I/O operations running.
Impact: High
Customer circumvention: Avoid running setvv on these volumes.
Customer recovery steps: None.


Not sure if "uncompressed volumes" means tpvv or what, but better safe than sorry

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 


Who is online

Users browsing this forum: MSN [Bot] and 37 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt