Remote Copy Scripts for Windows-based Oracle

[aberus]

Preface:
I'm not looking for advice on how to do this but comments on making things better is always welcome. I'm also not looking for anyone's blessings on my design or how it is deployed, this is the best I could come up with and works great for me. My sole reason for sharing is the lack of information existing on deploying Remote Copy and HPE support's complete lack of understanding and/or willingness to help and just wanting to engage in an expensive and useless services engagement.

My Environment:
-Source 3PAR's: 7450 and 8440
-Destination 3PAR: 8440
-Operating Systems is W2k8R2 and W2k12R2
-Because we are running Windows we are unable to use HP's Oracle Agent
-Oracle 12c Enterprise (4 servers totaling 32TB)
-Dedicated Remote Copy Groups created for each server containing all of the LUN's for consistency
-Remote Copy is configured as Periodic Async with all 4 RCIP ports active on each array
-Connection between data centers is dedicated 1GB fibre (not dark)

Core Requirements for Replication:
-Need to place the databases into hot backup mode prior to the RC sync
-Need to take the databases out of hot backup mode after the RC snapshots have been completed
-Need to log all activities
-Need to email success and failure notifications
-Need to easily control the schedule
-Need to allow non-SAN people to manually run this on demand

Prerequisites:
-Created encrypted password file for 3PAR auth
-Note: I have a dedicated Windows VM where SSMC and 3PAR CLI is installed and this is also where my Scheduled Tasks are configured to run
-Created the following folder structure on the SSMC server: "D:\3parlogs\RemoteCopy\Archive"
-SQL scripts written to put the database into and take it out of hot backup mode
-Scheduled task created on the destination server (DB server) where credentials are specified to run the start and end hot backup mode scripts.

Script Flow Overview:
-Scheduled tasks is configured to run every 6 hours
-Launches a batch file named "<servername>_rcg.cmd"
-Set script specific environment variables
-Set 3PAR variables:
-3PAR Name
-Encrypted password file path
-Set System Variables
-Source server name
-Remote copy group name
-Set Email Variables:
-SMTP IP/CNAME/Servername
-From address (SSMC Server)
-To addresses (email or DL)

***Note: THE ABOVE VAIABLES ARE THE ONLY SETTINGS THAT NEED TO BE SET FOR THIS SCRIPT TO RUN***

-Pipe variables to log file (assists with troubleshooting)
:startrcopy
-Start the RCG
-Pipe out the RCG status
-Delete the Oracle trigger files on the source server showing before and after delete operation
:starthotbackup
-Launch the start hot backup mode scheduled tasks on the source server (DB)
-if the scheduled task fails it will abort the script and send an email
:rcsync
-Launch the RC sync operation on the 3PAR
:endhotbackup
-Launch the end hot backup mode scheduled tasks on the source server (DB)
-if the scheduled task fails it will abort the script and send an email
:rcsyncstatus
-The "showrcopy groups <rcg_anme>" command is piped to a txt file which this routine will check every 30 seconds, as soon as the log no longer contains the word "Syncing" it will complete and send out the success email.

[MammaGutt]

I know you wasn't looking for anyones blessing, but replication is not backup.

Just out of curiosity, why don't you just dump the DB and copy it off the server? Not quite sure I understand what disasters you are trying to have a solution for and how you would make use of this replication.

Seems like you are looking for something which Remote Copy isn't really made to do. Am I understanding this right that you've actually configured the rcopygroups as sync and just start and stop them to force an "async replication" when you are ready for it? And in most of the time the RC-group is just stopped?

Have you considered Recovery Manager for Oracle/RMC and create snapshots for this? Maybe sync repliation between the arrays and RM-O/RMC on one or both of the system. If your 8440 is rather new, then it has the All-Inclusive licensing which includes those licenses.

[aberus]

Thank you for the reply but I think you lost me, I lost you or maybe I didn't explain very well our situation. This is not a backup, I use CommVault for that purpose and it works well.

The disaster that this solution is for is if my primary data center fell off the map, I would then transfer all operations over to our disaster recovery data center which is about 300 miles away. It was mandated by my management that they do not wish this to be synchronous and wanted at least a 6 hours difference between the systems. This provides further protection for viruses, maleware, cryptolocker, etc. This initial version of the script does not cover this requirement because I am triggering it to run every 6 hours but will eventually modify it to run once per hour keeping only 6 snapshots. I haven't work though all of the logic on that yet but it will most likely look something like this:
-When the script runs, and after setting the environment variables it will create a virtual copy of the VV at the disaster recovery DC. The name of these snapshots will be predetermined for obvious reasons.
-it will then delete the oldest snapshot
-Then it will continue on with the script

I'm not sure what you mean when you say that this is not what remote copy is meant for, with my conversations with HPE and from the user guide this is exactly what it is meant do to, the only twist I have is that the 3PAR scheduling capabilities doesn't allow us to do what we need nor is it capable of intelligent communication with Oracle on Windows so I am forced to improvise which is what I have done here. I have several feature requests in and support has "grudgingly" agreed that this works, plus I have tested it exhaustively with my DBA's and SAP application teams and it worked well.

I came from the EVA world using RSM, this is pretty much the same process I did with RSM except for the EVA HP actually had a solution, they do not for the 3PAR.

I am unable to use Recovery Manager for Oracle, as I stated in my post it is not available on the Windows platform therefore it is not an option which is why I am in this position to begin with. When working with our local sales rep prior to the purchase this was discussed in detail, and we just learned a couple of months ago when we went to deploy that recovery manager would not work for Windows, when I pushed back on my sales guy he said that it did, then a few days later confirmed that it didn't. He bought me lunch though so its all good.

With Oracle you can't just replicate the database without putting it into backup mode, you have to freeze the headers or it will come up 100% of the time as corrupt and will require a restore from backup (ComVault), the point of this is to avoid that lengthy process. So instead what I do is run the process that I laid out and in addition to that I am synchronously pushing archive logs, via Oracle, to the recovery site to further minimize dataloss.

Now it is a different story on my SQL servers, I am able to use Recovery Manager for this, although it is still not as flexible as I require it to be.

The remote copy group is not in a stopped state, I put that logic in the script to cover an event in which it may be in a stopped state, for whatever reason. I'm the lead SAN guy here but not the only one who manages it so it's a safety net.

For my VMware environment I use Zerto, and it work splendidly!

I am fully licensed on all of my 8000 series arrays.

[MammaGutt]

Wow, you are right. RMC for Oracle doesn't support Windows! That's just ridiculous.

If the rcopy groups aren't configured for sync and stopped, how did you configure it to be able to ensure that the Oracle database is in hot backup mode when the replication starts/is running?

[aberus]

If you look at the script flow overview in my original post you will see a section starthotbackup, this is where I'm putting it into backup mode which completes before the rc sync operation. If you look at the script I attached it may make more sense.