Oracle Tips and Techs

Sunday, August 23, 2015

Oracle GoldenGate simple hands-on demo

I was working on a project where it involves use of Oracle GoldenGate for data migration from one database to another. We can term this project as cross platform data replication (migration and upgrade if done for whole database) as both the source and target system and database versions are different to each other.
I would like to show a simple demo on setting up Oracle GoldenGate software and configuring the same to use for migration or replication of data between databases (Oracle in our case) where as Oracle GoldenGate can be used for heterogeneous platform as well.

The post consists of the following
1. Environment details
2. GoldenGate installation
3. Prerequisites and Oracle GoldenGate software set up
4. Enabling TRANDATA
5. Param files set up
6. The GoldenGate process (Commands and Explanations)
7. Real time sample demo

Now let's get into the details.

1. Environment details

Source

Oracle 9.2.0.7.0 on SunOS operating system

OGG_source $ uname -sr
SunOS 5.10

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 – Production

Destination

Oracle 11.2.0.4 on Linux 2.6.32-358.0.1.el6.x86_64

OGG_target $ uname -sr
Linux 2.6.32-358.0.1.el6.x86_64

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

GoldenGate software used

Source: Oracle GoldenGate Version 11.1.1.1 13

Destination: Oracle GoldenGate Version 11.2.1.0.17

2. GoldenGate Installation

For GoldenGate installation please see Oracle GoldenGate Installation

3. Prerequisites and Oracle GoldenGate software set up

3.1 Prerequisites
As a prerequisite for enabling and utilizing OGG, archive log should be enabled in the source database. If bidirectional, both the database should run in ARCHIVELOG mode.

For enabling Archivelog mode, see Enabling Archivelog

Once Archivelog has been enabled, enable supplemental logging in source by issuing the below command

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

3.2 OGG software setup
The following has to be performed on source and target databases to set up GG owner and create/grant required privileges and roles required to perform GG activities.
On Source:

export ORACLE_BASE=/r01/app/oracle/
export ORACLE_HOME=/r01/app/oracle/product/9.2.0
export PATH=$PATH:$ORACLE_HOME/bin:/backup/PRNETSDB/ogg/11.1
export ORACLE_SID=PRNETSDB
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export SHLIB_PATH=$ORACLE_HOME/lib
export GGH=/backup/PRNETSDB/ogg/11.1

$ cd $GGH
$ ggsci

ggsci> create subdirs

All the GoldenGate required directories will be created under GoldenGate Home directory

SQL> create user ggs_dba identified by ggs_dba default tablespace TOOLS temporary tablespace TEMP profile DEFAULT;
SQL> grant CONNECT to ggs_dba;
SQL> grant SELECT ANY TABLE to ggs_dba; -- This can be restricted to tables that are only involved
SQL> grant ALTER SESSION to ggs_dba;

Change to the OGG HOME (GGH) directory and execute the scripts.

cd $GGH

-- Run the following as SYSDBA user with input "GGS_DBA"

SQL> @marker_setup.sql -- creates GGS_DDL_HIST table under GGS_DBA for DDL replication
SQL> @ddl_setup.sql -- To support DDL replication
SQL> @role_setup.sql
SQL> grant GGS_GGSUSER_ROLE to ggs_dba;
SQL> @ddl_enable.sql -- Enables GGS trigger

Change to dirprm directory which was created earlier and perform the below

cd $GGH/dirprm
vi mgr.prm
PORT 15000
DYNAMICPORTLIST 15010-15020
USERID GGS_DBA, PASSWORD ggs_dba

ggsci> start manager

ggsci>

On Target:

export ORACLE_BASE=/ne01/app/oracle
export ORACLE_HOME=/ne01/app/oracle/product/11.2.0.4
export PATH=$PATH:$ORACLE_HOME/bin:/backup/gg_kt/ogg/11.2
export ORACLE_SID=STNETSDB
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/backup/gg_kt/ogg/11.2
export SHLIB_PATH=$ORACLE_HOME/lib
export GGH=/backup/gg_kt/ogg/11.2

$ cd GGH
$ ggsci

ggsci> create subdirs

All the GoldenGate required directories will be created under GoldenGate Home directory

SQL> create user ggs_dba identified by ggs_dba default tablespace TOOLS temporary tablespace TEMP profile DEFAULT;
SQL> grant CONNECT to ggs_dba;
SQL> grant SELECT ANY TABLE to ggs_dba; -- This can be restricted to tables that are only involved
SQL> grant ALTER SESSION to ggs_dba;
-- Grant insert, update, delete privilege on tables replicated
SQL> grant insert, delete, update on [list of tables] to ggs_dba;

Or grant Oracle GoldenGate user DBA role till the migration activity is complete and then revoke the grant if required.

Change to the OGG HOME (GGH) directory and execute the scripts.

cd $GGH

Run the following as SYSDBA user with input "GGS_DBA"

SQL> @marker_setup.sql -- creates GGS_DDL_HIST table under GGS_DBA for DDL replication
SQL> @ddl_setup.sql -- To support DDL replication
SQL> @role_setup.sql
SQL> grant GGS_GGSUSER_ROLE to ggs_dba;
SQL> @ddl_enable.sql -- Enable GGS trigger

Change to dirprm directory which was created earlier and perform the below

cd $GGH/dirprm
vi mgr.prm
PORT 12000
DYNAMICPORTLIST 12010-12020
USERID GGS_DBA, PASSWORD ggs_dba
cd $GGH
ggsci
ggsci 1> start manager
ggsci 2> edit params ./GLOBALS -- This creates a vi file named GLOBALS* under the current directory
GGSCHEMA ggs_dba 
CHECKPOINTTABLE ggs_dba.checkpoint
:wq – to save the file
ggsci 3> dblogin userid ggs_dba Password ggs_dba
ggsci 4> add checkpointtable ggs_dba.checkpoint

Enable ENABLE_GOLDENGATE_REPLICATION process as this is a mandatory step if the database is of Oracle version 11g and above.

SQL> ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION = TRUE SCOPE=BOTH;

* To learn more about GLOBALS and other parameter files, refer this oracle documentation link

4. Enabling TRANDATA

Enabling TRANDATA (transaction data) is explained in Enable TRANDATA
Once the TRANDATA is enabled for individual tables, then the parameter files can be created

5. Param files set up

All the parameters set up required for our demo is given in this link Simple parameter files for Oracle GoldenGate
Upon completion of Parameter files creation, we can start the Oracle GoldenGate replication process.

6. The GoldenGate process (Commands and Explanations)

Please see this link OGG step by step process for the Oracle GoldenGate commands, step by step processes and their simple explanations.
Thus we have completed our Oracle GoldenGate set up and replication.

7. Real time sample demo

As we have completed the Oracle GoldenGate replication between 2 databases of different versions and platforms, the real time demo shown will be of much useful in understanding the process better.

Please see here for the demo - Oracle GoldenGate sample demo.

Happy Replicating!!

Selvakumar Nagulan

A person with hard working ability, will power and passion to finish what he starts. An Oracle database administrator trying to explore every nook and corner possible in Oracle technology. A blogger, a story and poem writer at leisure. A guy who maintain at most patience even when the whole world rejects him.

OGG sample real time demo

This post gives a real time demo of Oracle GoldenGate replication. For complete installation and set up, see this link

Terminologies used
OGG_source $ -- refers commands executed from source server
OGG_target $ -- refers commands executed from target server
SQL@OGG_source > -- refers commands executed from source SQLPLUS terminal
SQL@OGG_target > -- refers commands executed from target SQLPLUS terminal
GGSCI (source) -- refers commands executed from source GGSCI terminal
GGSCI (target) -- refers commands executed from target GGSCI terminal

At line 69, the initial extract is started from OS as a background process. The reason being the initial extract might run for hours depending on the amount of data it has to unload from the source database and hence to let it run without any issues till it completes, we are starting the process as a background process.
Also we are grepping the start time of the extract at line 74 to use it at a later time in the replication operation. See line 245 where this captured time is used.

OGG_source $ cd $GGH
OGG_source $ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.1.1.1 13067027
Solaris, sparc, 64bit (optimized), Oracle 9i on Oct 14 2011 08:02:21

Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.



GGSCI (source) 1> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING


GGSCI (source) 2> ADD EXTRACT DEMO1E, TRANLOG, THREADS 2, BEGIN NOW
EXTRACT added.


GGSCI (source) 3> ADD EXTTRAIL /backup/PRNETSDB/ogg/11.1/dirdat/ea, EXTRACT DEMO1E, Megabytes 200
EXTTRAIL added.


GGSCI (source) 4> start extract DEMO1E

Sending START request to MANAGER ...
EXTRACT DEMO1E starting


GGSCI (source) 5> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DEMO1E      00:00:07      00:00:06


GGSCI (source) 6> ADD EXTRACT DEMO1P, EXTTRAILSOURCE /backup/PRNETSDB/ogg/11.1/dirdat/ea
EXTRACT added.


GGSCI (source) 7> ADD RMTTRAIL /backup/gg_kt/ogg/11.2/dirdat/pa, EXTRACT DEMO1P, Megabytes 100
RMTTRAIL added.


GGSCI (source) 8> start extract DEMO1P

Sending START request to MANAGER ...
EXTRACT DEMO1P starting


GGSCI (source) 9> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DEMO1E      00:00:00      00:00:06
EXTRACT     RUNNING     DEMO1P      00:00:00      00:00:22


GGSCI (source) 10> ADD EXTRACT DEMO1I, sourceistable
EXTRACT added.


GGSCI (source) 11> exit
OGG_source $ nohup extract paramfile $GGH/dirprm/DEMO1I.prm reportfile $GGH/dirrpt/DEMO1I.rpt > DEMO1I.out 2>&1 &
[1]     17810
OGG_source $
[1] +  Done                    nohup extract paramfile $GGH/dirprm/DEMO1I.prm reportfile $GGH/dirrpt/DEMO1I.rpt > DEMO1I.out 2>&1 &
OGG_source $ cd dirrpt
OGG_source $ more DEMO1I.rpt|grep "Starting at"
                    Starting at 2015-08-07 08:02:26
OGG_source $ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Fri Aug 7 08:03:15 2015

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production

SQL@OGG_source > select count(*) from sdas.ogg_test;

  COUNT(*)
----------
      7004

********** Operation on Target system starts here **********

SQL@OGG_target > select count(*) from sdas.tgt_ogg_test;

  COUNT(*)
----------
         0

SQL@OGG_target > !
OGG_target $ cd $GGH
OGG_target $ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.17 17888650 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131215.2328_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Dec 16 2013 03:43:25

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.



GGSCI (target) 1> add rep TDEMO1I, exttrail /backup/gg_kt/ogg/11.2/dirdat/ia, nodbcheckpoint
REPLICAT added.


GGSCI (target) 2> start replicat TDEMO1I

Sending START request to MANAGER ...
REPLICAT TDEMO1I starting


GGSCI (target) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
REPLICAT    RUNNING     TDEMO1I     00:00:00      00:00:01


GGSCI (target) 4> stats replicat TDEMO1I

Sending STATS request to REPLICAT TDEMO1I ...

Start of Statistics at 2015-08-07 08:08:46.

Replicating from SDAS.OGG_TEST to SDAS.TGT_OGG_TEST:

*** Total statistics since 2015-08-07 08:08:29 ***
        Total inserts                                   7004.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                7004.00

*** Daily statistics since 2015-08-07 08:08:29 ***
        Total inserts                                   7004.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                7004.00

*** Hourly statistics since 2015-08-07 08:08:29 ***
        Total inserts                                   7004.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                7004.00

*** Latest statistics since 2015-08-07 08:08:29 ***
        Total inserts                                   7004.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                7004.00

End of Statistics.


GGSCI (target) 5> send rep TDEMO1I, getlag

Sending GETLAG request to REPLICAT TDEMO1I ...
Last record lag: 359 seconds.
At EOF, no more records to process.


GGSCI (target) 6> stop rep TDEMO1I

Sending STOP request to REPLICAT TDEMO1I ...
Request processed.


GGSCI (target) 7> exit
OGG_target $ exit
exit

SQL@OGG_target > select count(*) from sdas.tgt_ogg_test;

  COUNT(*)
----------
      7004

********** Insert and update some values in the source ************

SQL@OGG_source > insert into sdas.ogg_test values (20202,'Selvakumar Nagulan',sysdate,'N',20202);

1 row created.

SQL@OGG_source > update sdas.ogg_test set emp_name='Priyanka' where emp_id=7000;

1 row updated.

SQL@OGG_source > commit;

Commit complete.

SQL@OGG_source > select count(*) from sdas.ogg_test;

  COUNT(*)
----------
      7005

*********** Check the changes in target which still didn't reflect here ***********
*********** Do the rest of OGG operations to enable real time sync ****************

SQL@OGG_target > select * from sdas.tgt_ogg_test where emp_id=7000;

EMP_ID     EMP_NAME                                           JOINDATE  S     SALARY
---------- -------------------------------------------------- --------- - ----------
7000       Priya                                              20-MAY-15 A       7000

SQL@OGG_target > select * from sdas.tgt_ogg_test where emp_id=20202;

no rows selected

SQL@OGG_target > !
OGG_target $ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.17 17888650 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131215.2328_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Dec 16 2013 03:43:25

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.



GGSCI (target) 1> dblogin userid ggs_dba password ggs_dba
Successfully logged into database.

GGSCI (target) 2> ADD REPLICAT DEMO1R, EXTTRAIL /backup/gg_kt/ogg/11.2/dirdat/pa, checkpointtable ggs_dba.CHECKPOINT
REPLICAT added.


GGSCI (target) 3> ALTER REPLICAT DEMO1R, BEGIN 2015-08-07 08:02:26
REPLICAT altered.


GGSCI (target) 4> start replicat DEMO1R

Sending START request to MANAGER ...
REPLICAT DEMO1R starting


GGSCI (target) 5> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
REPLICAT    RUNNING     DEMO1R      00:06:19      00:00:08
REPLICAT    STOPPED     TDEMO1I     00:00:00      00:02:48


GGSCI (target) 6> stats rep DEMO1R

Sending STATS request to REPLICAT DEMO1R ...

Start of Statistics at 2015-08-07 08:12:03.

Replicating from SDAS.OGG_TEST to SDAS.TGT_OGG_TEST:

*** Total statistics since 2015-08-07 08:11:44 ***
        Total inserts                                      1.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   2.00

*** Daily statistics since 2015-08-07 08:11:44 ***
        Total inserts                                      1.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   2.00

*** Hourly statistics since 2015-08-07 08:11:44 ***
        Total inserts                                      1.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   2.00

*** Latest statistics since 2015-08-07 08:11:44 ***
        Total inserts                                      1.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   2.00

End of Statistics.


GGSCI (target) 7> send rep DEMO1R, ggetlag

Sending GGETLAG request to REPLICAT DEMO1R ...

ERROR: unknown request.

2015-08-07 08:12:28  WARNING OGG-01742  Command sent to REPLICAT DEMO1R returned with an ERROR response.


GGSCI (target) 8> send rep DEMO1R, getlag

Sending GETLAG request to REPLICAT DEMO1R ...
Last record lag: 379 seconds.
At EOF, no more records to process.


GGSCI (target) 9> stop rep DEMO1R

Sending STOP request to REPLICAT DEMO1R ...
Request processed.


GGSCI (target) 10> exit
OGG_target $ exit
exit

SQL@OGG_target > select count(*) from sdas.tgt_ogg_test;

  COUNT(*)
----------
      7005

SQL@OGG_target > select * from sdas.tgt_ogg_test where emp_id=7000;

EMP_ID     EMP_NAME                                           JOINDATE  S     SALARY
---------- -------------------------------------------------- --------- - ----------
7000       Priyanka                                           20-MAY-15 A       7000

SQL@OGG_target > select * from sdas.tgt_ogg_test where emp_id=20202;

EMP_ID     EMP_NAME                                           JOINDATE  S     SALARY
---------- -------------------------------------------------- --------- - ----------
20202      Selvakumar Nagulan                                 07-AUG-15 N      20202

SQL@OGG_target > !
OGG_target $ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.17 17888650 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131215.2328_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Dec 16 2013 03:43:25

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.



GGSCI (target) 1> edit params DEMO1R

********* Here remove the HANDLECOLLISIONS line from the param file and start the replicat **********

GGSCI (target) 2> start rep DEMO1R

Sending START request to MANAGER ...
REPLICAT DEMO1R starting


GGSCI (target) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
REPLICAT    RUNNING     DEMO1R      00:00:00      00:00:06
REPLICAT    STOPPED     TDEMO1I     00:00:00      00:05:50


GGSCI (target) 4> exit
OGG_target $ exit
exit

********* The source and target are now in sync with real time replication enabled **********

SQL@OGG_source > insert into sdas.ogg_test values (30303,'Rafael Nadal',sysdate,'N',30303);

1 row created.

SQL@OGG_source > commit;

Commit complete.

SQL@OGG_source >

SQL@OGG_target >
SQL@OGG_target > select * from sdas.tgt_ogg_test where emp_id=30303;

EMP_ID     EMP_NAME                                           JOINDATE  S     SALARY
---------- -------------------------------------------------- --------- - ----------
30303      Rafael Nadal                                       07-AUG-15 N      30303

SQL@OGG_target >

Happy Replicating!!

Selvakumar Nagulan

Sunday, August 23, 2015

Oracle GoldenGate simple hands-on demo

OGG sample real time demo

Oracle Certified Specialist

Microsoft Certified: Azure Fundamentals

Oracle Certified Associate

Total Pageviews