Monday, December 20, 2021

Oracle Application/session tracing methods - 2

 In my previous post, we saw how to trace individual sessions. The challenge involved in real world performance issues is that mostly there won't be any single session that would run in the DB but the application would spawn multiple processes/sessions. So in this case if we want to trace a session, it will be tedious task to identify which session to trace. Many time there would be run away sessions and a few times even the developers wouldn't be knowing the entire work flow which will again add complexity to the existing issue. 


During these times, instead of tracing individual session we can go ahead and trace set of sessions using application tracing methods. In this post, we will look at the methods on how to do the application tracing

Application level tracing

Application tracing can be performed by using 

  • DBMS_SQL_MONITOR
  • DBMS_MONITOR

We will have a separate post on DBMS_SQL_MONITOR later and now will look into the usage of DBMS_MONITOR

DBMS_MONITOR

Information that we need to proceed with application tracing using dbms_monitor are either of the 3 below

  • Client info
  • Module
  • Action
All of the 3 might or might not be obtained from v$session as v$session could also contain null for these fields. For eg:
SQL> set sqlformat ansiconsole
SQL> select sid, serial#, username, client_info, module, action from v$session where username not in ('SYS','SYSTEM')
SID   SERIAL#  USERNAME  CLIENT_INFO  MODULE                                          ACTION
---------------------------------------------------------------------------------------------
17    63073    SOE       Swingbench Load Generator  JDBC Thin Client
401   47594    SOE       Swingbench Load Generator  JDBC Thin Client
778   11025    SOE                                  SQL*Plus
780   55219    SYSRAC                               oraagent.bin@linux75-2.selvapc.com (TNS V1-V3)
1151  4268     SOE       Swingbench Load Generator  JDBC Thin Client
1157  61639    SYSRAC                               oraagent.bin@linux75-2.selvapc.com (TNS V1-V3)
1158  10873    TEST                                 SQL Developer
1166  11417    SOE       Swingbench Load Generator  JDBC Thin Client


8 rows selected.

SQL> 
You can see a few session has CLIENT_INFO and a few doesn't and none of sessions has info on ACTION field. 

We can use DBMS_APPLICATION_INFO package to extract the details needed or to set them as well by using READ_CLIENT_INFO, READ_MODULE, SET_ACTION, SET_CLIENT_INFO or SET_MODULE procedures which will be explained below

READ_CLIENT_INFO and READ_MODULE can be used to read the details from the specific SID using v$session view
SET_ACTION, SET_CLIENT_INFO or SET_MODULE can be used to set the names for the specific program or piece of code which makes tracing easy. 
The below code provides an example of how to insert the DBMS_APPLICATION_INFO package in application program code making way for us to trace as desired. 
CREATE
	OR replace PROCEDURE up_test1 AS

BEGIN
	dbms_application_info.set_module(module_name=> 'Change name', action_name=> 'insert id');

	INSERT INTO test1 (id,name)
	VALUES (102,'ORABLISS');

	dbms_application_info.set_action(action_name=> 'update id');

	UPDATE test1
	SET id = 101
	WHERE name = 'test';

	dbms_application_info.set_module(NULL, NULL);
END;
/
You can see I have inserted DBMS_APPLICATION_INFO.SET_MODULE in 2 places one with module_name and action_name and another with only action_name. This way we can trace the parts only we need in code with large number of lines. 

Once we introduced the module_name or action_name we can trace the sessions running this piece of code without even knowing the session id of the sessions executing them. 

Use DBMS_MONITOR.SERV_MOD_ACT_TRACE_ENABLE procedure to trace sessions as desired for investigation

For tracing sessions using module_name:
SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_ENABLE(service_name=>'ORCL',module_name=>'Change name');

PL/SQL procedure successfully completed.

-- Run the procedure using other session which will trigger the tracing 
-- Once the procedure is completed we can disable trace as follows. 

SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_disable(service_name=>'ORCL',module_name=>'Change name');

PL/SQL procedure successfully completed.

SQL> 

For tracing sessions performing specific action:

we can also specify the action_name which will trigger tracing only when the specific action name is started in the program
SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_ENABLE(service_name=>'ORCL',module_name=>'Change name',action_name=> 'update id',waits=>true,binds=>true);

PL/SQL procedure successfully completed.

-- Run the procedure using other session which will trigger the tracing 
-- This time tracing will capture only the update statement in the procedure as the action is named as 'update id'
-- Once the procedure is completed we can disable trace as follows. 

SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_disable(service_name=>'ORCL',module_name=>'Change name',action_name=> 'update id');

PL/SQL procedure successfully completed.

SQL> 
The above SERV_MOD_ACT_TRACE_ENABLE procedure follows strict hierarchy, meaning - service_name is mandatory followed by module_name and action_name. If no module_name is provided, we can't provide action_name and if no action_anme is provided all the action under the module will be traced. 

For tracing sessions using specific service_name:

If we don't specify any module_name, all the sessions connecting to the DB using the service_name will be traced. Different applications can connect to same database using different service names as desired by the database admin to segregate them. 
SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_ENABLE(service_name=>'ORCL');

PL/SQL procedure successfully completed.

-- This time tracing will capture all the sessions that connect to DB using ORCL service
-- We can disable trace as follows. 

SQL> exec dbms_monitor.SERV_MOD_ACT_TRACE_DISABLE(service_name=>'ORCL');

PL/SQL procedure successfully completed.

SQL> 

For tracing sessions using Client_identifier:


v$session will have client_info filed populated for most of the application. If it is null like the example I gave in this post, we can use the DBMS_APPLICATION_INFO.SET_CLIENT_INFO procedure to set in the program/main procedure similar to below
CREATE
	OR replace PROCEDURE up_test2 AS

BEGIN
	dbms_application_info.set_client_info('client info test');
	dbms_session.set_identifier('client info test');

	INSERT INTO test1 (id,name)
	VALUES (102,'ORABLISS');

	UPDATE test1
	SET id = 101
	WHERE name = 'test';
	
	dbms_session.clear_identifier;
END;
/
As I have now included the DBMS_SESSION.SET_IDENTIFIER, we can trace the session without searching for the session id. 
Setting only CLIENT_INFO will be useful to find the session details to enable tracing and for automatic tracing of session with client id, we need the dbms_session.set_identifier
SQL> exec DBMS_MONITOR.CLIENT_ID_TRACE_ENABLE('client info test');

PL/SQL procedure successfully completed.

-- Run the procedure up_test2 using other session which will trigger the tracing using client identifier
-- Once the procedure is completed we can disable trace as follows (optional)

SQL> exec DBMS_MONITOR.CLIENT_ID_TRACE_disable('client info test');

PL/SQL procedure successfully completed.

SQL>

Database level tracing

The above methods explained regarding the options of tracing parts of database and its sessions. 
If a need arise to trace the entire database (I don't think any one would do this on a production database), we can do so by DBMS_MONITOR.DATABASE_TRACE_ENABLE procedure
SQL> exec DBMS_MONITOR.DATABASE_TRACE_ENABLE;

PL/SQL procedure successfully completed.

-- All the sessions in the database will be traced
-- waits, binds, instance_name and plan_stat can be provided as arguments

-- To disable tracing

SQL> exec DBMS_MONITOR.DATABASE_TRACE_DISABLE;

PL/SQL procedure successfully completed.

SQL>

With this, we have looked at all the ways we can trace the application/DB sessions. 
All the trace files generated will be in Oracle proprietary format. To convert the trace files generated and make use of the trace files, check this link on tkprof


Happy Tracing

Oracle Application/session tracing methods - 1

 Tracing forms an important method when it comes to database and/or SQL performance tuning. In this blog post let's see the various methods we can trace a session or an application or an entire database and how to read the trace files generated 

Types of tracing: 

  • Session level
  • Application level
  • Database level


Prerequisites

  • Check timed_statistics value is TRUE;
  • Check statistics_level is typical or all
  • Set max_dump_file_size to unlimited so that the trace is not abruptly ended due to file size reached

Example tracing own session: 

When we need to trace the same session that we are connected to trouble shoot a query, we can perform the below

SQL_TRACE

[oracle@linux75-2 ~]$ sql soe/soe

SQLcl: Release 12.2.0.1.0 RC on Thu Dec 02 23:53:32 2021

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


SQL> show parameter timed_statistics
NAME             TYPE    VALUE
---------------- ------- -----
timed_statistics boolean TRUE

SQL> show parameter statistics_level
NAME             TYPE   VALUE
---------------- ------ -------
statistics_level string TYPICAL

SQL> show parameter max_dump_file_size
NAME               TYPE   VALUE
------------------ ------ ---------
max_dump_file_size string unlimited

SQL> alter session set tracefile_identifier='MyTrace';

Session altered.

SQL> alter session set sql_trace=true;

Session altered.

SQL> -- Run the query that needs to be traced now
SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> alter session set sql_trace=false;

Session altered.

SQL>

10046 tracing

We can also enable 10046 tracing to perform the same as below
SQL> -- set the identifier to find the trace file easily in the trace directory
SQL> alter session set tracefile_identifier='My10046';

Session altered.

SQL> alter session set events '10046 trace name context forever, level 12';

Session altered.

SQL> -- Run the query to be traced now
SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> alter session set events '10046 trace name context off';

Session altered.

SQL> show parameter diag
NAME            TYPE   VALUE
--------------- ------ ---------
diagnostic_dest string /u01/orcl
SQL> exit
For 10046 tracing, the following values are valid for trace level
  • 0 - Off
  • 2 - Similar to regular sql_trace
  • 4 - Same as 2, but with addition of bind variable values
  • 8 - Same as 2, but with addition of wait events
  • 12 - same as 2, but with both bind variable values and wait events

DBMS_SESSION package

Own session can also be traced using dbms_session package
SQL> alter session set tracefile_identifier='MySess';

Session altered.

SQL> exec DBMS_SESSION.SESSION_TRACE_ENABLE(waits=> true, binds=> true);

PL/SQL procedure successfully completed.

SQL> select count(*) from soe.addresses where country='India';

  COUNT(*)
----------
     19536

SQL> exec DBMS_SESSION.SESSION_TRACE_DISABLE();

PL/SQL procedure successfully completed.

SQL> exit
All the trace files will be available by default in the trace directory under diagnostic_dest. 
[oracle@linux75-2 ~]$ cd /u01/orcl/diag/rdbms/orcl/orcl/trace/
[oracle@linux75-2 trace]$ ls -lrt *My*
-rw-r-----. 1 oracle oinstall  1310 Dec  2 23:55 orcl_ora_8930_MyTrace.trm
-rw-r-----. 1 oracle oinstall  2560 Dec  2 23:55 orcl_ora_8930_MyTrace.trc
-rw-r-----. 1 oracle oinstall  2984 Dec  2 23:55 orcl_ora_8930_My10046.trm
-rw-r-----. 1 oracle oinstall 26449 Dec  2 23:55 orcl_ora_8930_My10046.trc
-rw-r-----. 1 oracle oinstall  3404 Dec  3 00:38 orcl_ora_12142_MySess.trm
-rw-r-----. 1 oracle oinstall 33729 Dec  3 00:38 orcl_ora_12142_MySess.trc
[oracle@linux75-2 trace]$ 
By default, the trace files will be generated by it's server process id. In the above, 8930 and 12142 are the process id of the sessions I logged into the database. Since there will be many processes running and by default oracle trace will be enabled for many background processes, we will be in a position that we are lost when searching for the trace file generated. To find the trace files easily, we are setting the tracefile_identifier as our first step of tracing. 

Example tracing another session:

In most circumstances as a DBA, we will not be tracing our own session but will have to another session whether a user is already logged in or is about to login. Let's take a look on how to we do this 

Start trace as soon as logon

We can create an after logon trigger to enable tracing for the session whenever a user logon to the database as below

CREATE OR REPLACE TRIGGER sys.set_trace AFTER LOGON ON DATABASE
WHEN ( user LIKE '&USERNAME' ) DECLARE
lcommand VARCHAR(200);
BEGIN
EXECUTE IMMEDIATE 'alter session set statistics_level=ALL';
EXECUTE IMMEDIATE 'alter session set max_dump_file_size=UNLIMITED';
DBMS_MONITOR.SESSION_TRACE_ENABLE (WAITS=> TRUE , BINDS=> TRUE);
END SET_TRACE ;
/

Note: The user on which the tracing should be enabled should explicitly have execute privilege on alter session and DBMS_MONITOR package. Otherwise the logon will fail due to insufficient privilege issue.

Start tracing when a session is identified for tracing

Many a times, when an issue is identified we will be in a situation where already a session is established in the database. So all we need to do is to get the details of SID and Serial# of the session to begin tracing of that session. Mostly these sessions are running with high cpu consumption or high waits or consuming high DB time. 
So once we figured out the SID and Serial#, the session trace can be done as below
SQL> -- Finding session id and serial # to enable trace
SQL> select username, sid, serial# from v$session where username='SOE';

USERNAME                              SID    SERIAL#
------------------------------ ---------- ----------
SOE                                   783      48974

SQL> -- Enable trace for the session
SQL> EXECUTE DBMS_MONITOR.SESSION_TRACE_ENABLE(SESSION_ID=>783, SERIAL_NUM=>48974, WAITS=>TRUE, BINDS=>FALSE);

PL/SQL procedure successfully completed.

SQL> -- Now all the activities on session 783,48974 will be traced
SQL> -- We can disable trace once we have confirmation from user he/she completed their work to investigate
SQL>
SQL> EXECUTE DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>783, SERIAL_NUM=>48974);

PL/SQL procedure successfully completed.

SQL> 
If the session is disconnected before we stop tracing, tracing will automatically be stopped and upon trying to stop the tracing, we will get the below error
BEGIN DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>7, SERIAL_NUM=>44); END;

*
ERROR at line 1:
ORA-00030: User session ID does not exist.
ORA-06512: at "SYS.DBMS_MONITOR", line 144
ORA-06512: at line 1

We can also use the DBMS_SYSTEM package to trace the session as below. DBMS_MONITOR explained above gives better control of what we need in terms of tracing
SQL> select username, sid, serial# from v$session where username='SOE';

USERNAME                              SID    SERIAL#
------------------------------ ---------- ----------
SOE                                  1151       6659

SQL> exec sys.dbms_system.set_sql_trace_in_session(1151, 6659, TRUE);

PL/SQL procedure successfully completed.

SQL> exec sys.dbms_system.set_sql_trace_in_session(1151, 6659, FALSE);

PL/SQL procedure successfully completed.

SQL> desc sys.dbms_system
...
...
PROCEDURE SET_SQL_TRACE_IN_SESSION
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SID                            NUMBER                  IN
 SERIAL#                        NUMBER                  IN
 SQL_TRACE                      BOOLEAN                 IN
...
...
SQL> 
So we now conclude with different methods of tracing a session. In the next post, we will take a look at Application tracing methods. Please check this link to learn on Application tracing methods

To convert raw trace file to human readable format, check this link on tkprof 

References

General SQL_TRACE / 10046 trace Gathering Examples (Doc ID 1274511.1)

Happy tracing...!!!