Monday, December 20, 2021

Oracle Application/session tracing methods - 1

 Tracing forms an important method when it comes to database and/or SQL performance tuning. In this blog post let's see the various methods we can trace a session or an application or an entire database and how to read the trace files generated 

Types of tracing: 

  • Session level
  • Application level
  • Database level


Prerequisites

  • Check timed_statistics value is TRUE;
  • Check statistics_level is typical or all
  • Set max_dump_file_size to unlimited so that the trace is not abruptly ended due to file size reached

Example tracing own session: 

When we need to trace the same session that we are connected to trouble shoot a query, we can perform the below

SQL_TRACE

[oracle@linux75-2 ~]$ sql soe/soe

SQLcl: Release 12.2.0.1.0 RC on Thu Dec 02 23:53:32 2021

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


SQL> show parameter timed_statistics
NAME             TYPE    VALUE
---------------- ------- -----
timed_statistics boolean TRUE

SQL> show parameter statistics_level
NAME             TYPE   VALUE
---------------- ------ -------
statistics_level string TYPICAL

SQL> show parameter max_dump_file_size
NAME               TYPE   VALUE
------------------ ------ ---------
max_dump_file_size string unlimited

SQL> alter session set tracefile_identifier='MyTrace';

Session altered.

SQL> alter session set sql_trace=true;

Session altered.

SQL> -- Run the query that needs to be traced now
SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> alter session set sql_trace=false;

Session altered.

SQL>

10046 tracing

We can also enable 10046 tracing to perform the same as below
SQL> -- set the identifier to find the trace file easily in the trace directory
SQL> alter session set tracefile_identifier='My10046';

Session altered.

SQL> alter session set events '10046 trace name context forever, level 12';

Session altered.

SQL> -- Run the query to be traced now
SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> alter session set events '10046 trace name context off';

Session altered.

SQL> show parameter diag
NAME            TYPE   VALUE
--------------- ------ ---------
diagnostic_dest string /u01/orcl
SQL> exit
For 10046 tracing, the following values are valid for trace level
  • 0 - Off
  • 2 - Similar to regular sql_trace
  • 4 - Same as 2, but with addition of bind variable values
  • 8 - Same as 2, but with addition of wait events
  • 12 - same as 2, but with both bind variable values and wait events

DBMS_SESSION package

Own session can also be traced using dbms_session package
SQL> alter session set tracefile_identifier='MySess';

Session altered.

SQL> exec DBMS_SESSION.SESSION_TRACE_ENABLE(waits=> true, binds=> true);

PL/SQL procedure successfully completed.

SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> exec DBMS_SESSION.SESSION_TRACE_DISABLE();

PL/SQL procedure successfully completed.

SQL> exit
All the trace files will be available by default in the trace directory under diagnostic_dest. 
[oracle@linux75-2 ~]$ cd /u01/orcl/diag/rdbms/orcl/orcl/trace/
[oracle@linux75-2 trace]$ ls -lrt *My*
-rw-r-----. 1 oracle oinstall  1310 Dec  2 23:55 orcl_ora_8930_MyTrace.trm
-rw-r-----. 1 oracle oinstall  2560 Dec  2 23:55 orcl_ora_8930_MyTrace.trc
-rw-r-----. 1 oracle oinstall  2984 Dec  2 23:55 orcl_ora_8930_My10046.trm
-rw-r-----. 1 oracle oinstall 26449 Dec  2 23:55 orcl_ora_8930_My10046.trc
-rw-r-----. 1 oracle oinstall  3404 Dec  3 00:38 orcl_ora_12142_MySess.trm
-rw-r-----. 1 oracle oinstall 33729 Dec  3 00:38 orcl_ora_12142_MySess.trc
[oracle@linux75-2 trace]$ 
By default, the trace files will be generated by it's server process id. In the above, 8930 and 12142 are the process id of the sessions I logged into the database. Since there will be many processes running and by default oracle trace will be enabled for many background processes, we will be in a position that we are lost when searching for the trace file generated. To find the trace files easily, we are setting the tracefile_identifier as our first step of tracing. 

Example tracing another session:

In most circumstances as a DBA, we will not be tracing our own session but will have to another session whether a user is already logged in or is about to login. Let's take a look on how to we do this 

Start trace as soon as logon

We can create an after logon trigger to enable tracing for the session whenever a user logon to the database as below

CREATE OR REPLACE TRIGGER sys.set_trace AFTER LOGON ON DATABASE
WHEN ( user LIKE '&USERNAME' ) DECLARE
lcommand VARCHAR(200);
BEGIN
EXECUTE IMMEDIATE 'alter session set statistics_level=ALL';
EXECUTE IMMEDIATE 'alter session set max_dump_file_size=UNLIMITED';
DBMS_MONITOR.SESSION_TRACE_ENABLE (WAITS=> TRUE , BINDS=> TRUE);
END SET_TRACE ;
/

Note: The user on which the tracing should be enabled should explicitly have execute privilege on alter session and DBMS_MONITOR package. Otherwise the logon will fail due to insufficient privilege issue.

Start tracing when a session is identified for tracing

Many a times, when an issue is identified we will be in a situation where already a session is established in the database. So all we need to do is to get the details of SID and Serial# of the session to begin tracing of that session. Mostly these sessions are running with high cpu consumption or high waits or consuming high DB time. 
So once we figured out the SID and Serial#, the session trace can be done as below
SQL> -- Finding session id and serial # to enable trace
SQL> select username, sid, serial# from v$session where username='SOE';

USERNAME                              SID    SERIAL#
------------------------------ ---------- ----------
SOE                                   783      48974

SQL> -- Enable trace for the session
SQL> EXECUTE DBMS_MONITOR.SESSION_TRACE_ENABLE(SESSION_ID=>783, SERIAL_NUM=>48974, WAITS=>TRUE, BINDS=>FALSE);

PL/SQL procedure successfully completed.

SQL> -- Now all the activities on session 783,48974 will be traced
SQL> -- We can disable trace once we have confirmation from user he/she completed their work to investigate
SQL>
SQL> EXECUTE DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>783, SERIAL_NUM=>48974);

PL/SQL procedure successfully completed.

SQL> 
If the session is disconnected before we stop tracing, tracing will automatically be stopped and upon trying to stop the tracing, we will get the below error
BEGIN DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>7, SERIAL_NUM=>44); END;

*
ERROR at line 1:
ORA-00030: User session ID does not exist.
ORA-06512: at "SYS.DBMS_MONITOR", line 144
ORA-06512: at line 1

We can also use the DBMS_SYSTEM package to trace the session as below. DBMS_MONITOR explained above gives better control of what we need in terms of tracing
SQL> select username, sid, serial# from v$session where username='SOE';

USERNAME                              SID    SERIAL#
------------------------------ ---------- ----------
SOE                                  1151       6659

SQL> exec sys.dbms_system.set_sql_trace_in_session(1151, 6659, TRUE);

PL/SQL procedure successfully completed.

SQL> exec sys.dbms_system.set_sql_trace_in_session(1151, 6659, FALSE);

PL/SQL procedure successfully completed.

SQL> desc sys.dbms_system
...
...
PROCEDURE SET_SQL_TRACE_IN_SESSION
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SID                            NUMBER                  IN
 SERIAL#                        NUMBER                  IN
 SQL_TRACE                      BOOLEAN                 IN
...
...
SQL> 
So we now conclude with different methods of tracing a session. In the next post, we will take a look at Application tracing methods. Please check this link to learn on Application tracing methods

To convert raw trace file to human readable format, check this link on tkprof 

References

General SQL_TRACE / 10046 trace Gathering Examples (Doc ID 1274511.1)

Happy tracing...!!!

Thursday, October 28, 2021

Cross platform 11g (Windows) to 19c (Linux) upgrade and migration using AutoUpgrade utility

Source and Target details are as given below


In this post, let's look at the procedure on how to migrate and upgrade the database version 11.2.0.4 running on Windows OS to database version 19.12 running on Linux OS (Exadata in this case) and plug into an existing CDB using AutoUpgrade utility. 

Step 0: Download AutoUpgrade utility from Doc ID 2485457.1 and install latest Java from Java Downloads
AutoUpgrade utility is java based and generic for all OS.

Step 1: Prepare config files for AutoUpgrade utility

Create a config file for analyze and fix up phase of the AutoUpgrade.

global.autoupg_log_dir=C:\Oracle4Win\AutoUpgrade\logs
#
# Database number 1 
#
upg1.dbname=TEST11G
upg1.start_time=NOW
upg1.source_home=C:\Oracle11g\product\11.2.0\dbhome_1
upg1.target_home=/tmp
upg1.sid=TEST11G
upg1.log_dir=C:\Oracle4Win\AutoUpgrade\logs
upg1.upgrade_node=localhost
upg1.target_version=19
upg1.restoration=no
See that I have mentioned target_home as /tmp as we will not perform the upgrade in source server.

Step 2: Run the analyze mode of AutoUpgrade

DB environment is as below:
set ORACLE_SID=TEST11G
set ORACLE_HOME=C:\Oracle11g\product\11.2.0\dbhome_1
set PATH=C:\Oracle11g\product\11.2.0\dbhome_1\bin;%PATH%
Command used is as below:
C:\oracle\product\19.0.0\jdk\bin\java -jar C:\Oracle4Win\AutoUpgrade\autoupgrade.jar -config C:\Oracle4Win\AutoUpgrade\test11g.cfg -mode analyze
Note: Don't confuse with the 19c patch I'm using for java as I have the latest java needed in the same source server in the 19c home. You can download and install latest java without 19c on source. Actual analyze is done on the 11g database with 11g home

We can use lsj command within the utility to list the jobs with Summary


Step 3: Check status file for any action to be taken.

Though there are no errors in the status file, Oracle recommendations are to be followed for optimum upgrade and post upgrade performance. We can click on Precheck Report to go through all the recommendations provided by the utility if needed.

C:\Oracle4Win\AutoUpgrade\logs\cfgtoollogs\upgrade\auto\status

Step 4: Start fixups mode of AutoUpgrade utility

Command used is as below:
C:\oracle\product\19.0.0\jdk\bin\java -jar C:\Oracle4Win\AutoUpgrade\autoupgrade.jar -config C:\Oracle4Win\AutoUpgrade\test11g.cfg -mode fixups

Let us now see a few snaps of PRECHECKS progress...



Let us now see a few snaps of PREFIXUPS progress...



The below command will details of the specific job which we are interested in.. 
status -job <jobnumber>



Step 5: Check status file for any action to be taken.

The prechecks passed and no manual intervention needed in our case. If we click on the Prechecks Report and you will see all the details of what the software has tested, whether any user action required, any errors, warnings reported and the informational messages as well.
Mostly, the fixup stage will fix issues like purging recycle bin, gathering stats prior to upgrade, suggesting parameter values removal/changes, suggest removal of deprecated components, etc., 
If there are errors reported, they need to be taken care before attempting upgrade. 

C:\Oracle4Win\AutoUpgrade\logs\cfgtoollogs\upgrade\auto\status


This concludes the AutoUpgrade utility activity on source server. We have now prepared our source database to be upgrade ready. 

Step 6: Make a consistent backup to avoid issues during upgrade.

Consistent backup is needed here as the source is Windows and target is Linux (Cross Platform) where recovery of windows archives on Linux is not supported.
  • Mount DB and trigger RMAN full backup
  • Transfer the Backup files and initSID file to target server

SQL> create pfile='C:\SharedPath\Backup\initTEST11G.ora' from spfile;
File created.

SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.

Total System Global Area  563691520 bytes
Fixed Size                  2283224 bytes
Variable Size             385878312 bytes
Database Buffers          167772160 bytes
Redo Buffers                7757824 bytes
Database mounted.
SQL> exit
C:\windows\system32>rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Mon Oct 4 14:15:37 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TEST11G (DBID=1234192708, not open)

RMAN> backup database format 'C:\SharedPath\Backup\%U' include current controlfile;

Starting backup at 04-OCT-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=156 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
…
….
…
channel ORA_DISK_1: starting piece 1 at 04-OCT-21
channel ORA_DISK_1: finished piece 1 at 04-OCT-21
piece handle=C:\SHAREDPATH\BACKUP\020APFE3_1_1 tag=TAG20211004T141619 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 04-OCT-21

RMAN> exit
Recovery Manager complete.
Step 7: Transfer the files to target

We can use copy/paste or scp or any method to transfer files needed for the restore on target server

The above step concludes all the activity that is needed on the source system. The following steps will be performed in the target system where the database will be upgraded to 19c. 

Step 8: Make an entry for TEST11G DB in oratab and Prepare init file

TEST11G database will have 19c home as the home path in the oratab file. 
initTEST11G.ora will be as below. 
[oracle@srv1 ~]$ more /u02/app/oracle/product/19.0.0.0/dbhome_2/dbs/initTEST11G.ora
*.audit_file_dest='/u02/app/oracle/product/19.0.0.0/dbhome_2/rdbms/audit'
*.audit_trail='db'
*.compatible='11.2.0.4.0'
*.control_files='+DATAC3/WNCCPD01/CONTROLFILE/test11g_control.ctl'
*.db_block_size=8192
*.db_domain=''
*.db_name='test11g'
*.db_recovery_file_dest='+RECOC3'
*.db_recovery_file_dest_size=4385144832
*.diagnostic_dest='/u02/app/oracle'
*.memory_target=3G
*.open_cursors=300
*.processes=300
*.remote_login_passwordfile='EXCLUSIVE'
*.undo_tablespace='UNDOTBS1'
*.db_create_file_dest='+DATAC3'
*.db_create_online_log_dest_1='+DATAC3'
[oracle@srv1 ~]$

Step 9: Restore Database

I have just provided the commands that needs to be run for restoring the database and not the output. We need to open the database in upgrade mode after restore as we are restoring 11g database under 19c home. 
## Mount the database
SQL> startup nomount


## Start RMAN and run the restore command
rman target /
run
{
restore controlfile from '/oradump/selva/TEST11G/020APFE3_1_1';
alter database mount;
catalog backuppiece '/oradump/selva/TEST11G/010APFDJ_1_1';
set newname for database to '+DATAC3';
set newname for tempfile 1 to ‘+DATAC3’;
restore database;
switch datafile all;
switch tempfile all;
}
RMAN> exit

## Rename the logfiles to proper location and open DB resetlogs in upgrade mode. 
sqlplus / as sysdba
SQL> select member from v$logfile;
SQL> alter database rename file 'C:\ORACLE11G\ORADATA\TEST11G\REDO03.LOG' to '+DATAC3/TEST11G/DATAFILE/REDO03.LOG';
SQL> alter database rename file 'C:\ORACLE11G\ORADATA\TEST11G\REDO02.LOG' to '+DATAC3/TEST11G/DATAFILE/REDO02.LOG';
SQL> alter database rename file 'C:\ORACLE11G\ORADATA\TEST11G\REDO01.LOG' to '+DATAC3/TEST11G/DATAFILE/REDO01.LOG';
SQL> alter database open resetlogs upgrade;
SQL> exit
Step 10: Prepare target config file

Upgrade configuration file will be as follows
[oracle@srv1]$ more upgtest.cfg
global.autoupg_log_dir=/home/oracle/logs
upg1.dbname=TEST11G
upg1.start_time=NOW
upg1.source_home=/u02/app/oracle/product/19.0.0.0/dbhome_2
upg1.target_home=/u02/app/oracle/product/19.0.0.0/dbhome_2
upg1.sid=TEST11G
upg1.log_dir=/home/oracle/logs
upg1.upgrade_node=localhost
upg1.target_version=19
upg1.restoration=no
upg1.target_cdb=WNCCPD018
upg1.target_pdb_name=UPGTEST
 
Note: I'm providing the target_pdb_name with a different name from the original name. Also note that we are not attempting to plug the database into an existing container database. 

Step11: Start upgrade mode of AutoUpgrade utility

Command will be as follows:  
java -jar /dcwnec_oradump/selva/xxx_TEST11G/autoupgrade.jar -config /dcwnec_oradump/selva/xxx_TEST11G/upgtest.cfg -mode upgrade

Let's see a few progress snaps. The upgrade process will take around 60 to 120 minutes depending upon various factors. So we can check the progress using lsj command as below








Detailed log will have the below entry..

All the below mentioned tasks will automatically be taken care. Config file should have details regarding keys if source is encrypted with TDE. Or the stage will error which then can be fixed manually by importing keys.


Few more status snaps before we conclude...




The upgrade process is completed and now we can see the PDB is created with the name we have provided in the config file. 



Step 12: Post Upgrade:

Run below query to identify PDB violations and fix accordingly.
column message format a50
column status format a9
column type format a9
column con_id format 9
column name for a30
select con_id, name, type, message, status from PDB_PLUG_IN_VIOLATIONS
where status<>'RESOLVED' order by time;

These violations will not stop the database from running normally as these are only WARNING messages but we need to take care of fixing the violations. 
If we get an ERROR message, we will not be able to open the PDB in read write mode which requires the violation to be fixed before opening the PDB in read write mode. 

With this, we are now concluding the upgrade exercise and the database is successfully upgraded from 11g running on windows to 19c running on Linux. 

References

Happy Upgrading...!!!