Showing posts with label 19c. Show all posts
Showing posts with label 19c. Show all posts

Tuesday, February 13, 2024

Connection to Oracle Autonomous Database

This article explains how to connect to an Oracle Autonomous database using SQL developer, SQL plus or other tools.

When we create an Oracle ADB on OCI, we just can't connect directly to the database even though the Network ACL is disabled. The reason is mTLS authentication is set to Required by default. 

Let us see how to connect to the ADB using SQL developer first and then SQL plus.

If you have downloaded SQL developer for connecting to your database on cloud, it's super easy to connect.

Method 1: Using mTLS  

Mutual TLS (mTLS) is a 2-way handshake between a client and a server, and a more secure way of connection compared to TLS.

Step 1: Download the wallet (This is required for mTLS)

Go to your Autonomous database details page => Click on Database connection => Click on Download wallet



Provide the password and click on Download. This is your wallet password and not the password for the database Admin user or the database application user you are trying to connect. 

Step 2: Once you have this credentials wallet zip, Open SQL Developer and edit the connection properties to connect to the Autonomous database as below



Make sure we have the connection type selected as Cloud Wallet and provide the downloaded zip file under configuration file. 
While selecting Service, we have different default service to choose from which differs on how the connection works. I have selected atp01_high of the below 5 available.
  • atp01_high
  • atp01_low
  • atp01_medium
  • atp01_tp
  • atp01_tpurgent
Click on the Test button to check the connectivity and we now have a Successful connection. 

How do we connect to using SQL Plus? 

To make a SQL Plus or any Oracle Call Interface (OCI), JDBC connection to the Autonomous database, we need the client software to be installed in the server from where we are trying to connect. 
Since we need the wallet location to be specified, we should have sqlnet.ora file and tnsnames.ora file

I'll use my existing on-prem Linux server to attempt a connection without the wallet file location specified in the sqlnet.ora but with only the tnsnames.ora edited with the connection details of the ADB. 
[oracle@linux-8 db-sample-schemas-main]$ cd order_entry/
[oracle@linux-8 order_entry]$ sqlplus admin/xxxxx@'(description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)
(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=g34efa58d1e9751_atp01_high.adb.oraclecloud.com))
(security=(ssl_server_dn_match=yes)))'

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Feb 7 03:56:14 2024
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

ERROR:
ORA-12506: TNS:listener rejected connection based on service ACL filtering


Enter user-name:
 
So where do I get the connection description information from? 
When I click on the Database connection button on the Database details page, I'll get the connection strings for all the 5 services or the downloaded zip will include the tnsnames.ora file containing this information. 

As you see when we didn't provide the wallet location, we are not able to connect with error ORA-12506. 

There are 2 things you can do to fix this. 
Easy way is to unzip the contents of the downloaded zip file to a directory on the server and define the TNS_ADMIN environment variable value to this directory and attempt the connection. Make sure to make changes in the sqlnet.ora file to denote the wallet location
[oracle@linux-8 db-sample-schemas-main]$ ls -lrt
total 28
-rwxrwx---. 1 root vboxsf 21984 Jan  9 21:17 Wallet_ATP01.zip
drwxrwx---. 1 root vboxsf  4096 Feb  7 04:00 db-sample-schemas-main
[oracle@linux-8 db-sample-schemas-main]$ unzip Wallet_ATP01.zip -d ./Wallet_ATP01
Archive:  Wallet_ATP01.zip
  inflating: ./Wallet_ATP01/ewallet.pem
  inflating: ./Wallet_ATP01/README
  inflating: ./Wallet_ATP01/cwallet.sso
  inflating: ./Wallet_ATP01/tnsnames.ora
  inflating: ./Wallet_ATP01/truststore.jks
  inflating: ./Wallet_ATP01/ojdbc.properties
  inflating: ./Wallet_ATP01/sqlnet.ora
  inflating: ./Wallet_ATP01/ewallet.p12
  inflating: ./Wallet_ATP01/keystore.jks
[oracle@linux-8 db-sample-schemas-main]$ cd Wallet_ATP01/
[oracle@linux-8 Wallet_ATP01]$ vi sqlnet.ora
[oracle@linux-8 Wallet_ATP01]$ more sqlnet.ora
WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY="?/network/admin")))
SSL_SERVER_DN_MATCH=yes
[oracle@linux-8 Wallet_ATP01]$ vi sqlnet.ora
[oracle@linux-8 Wallet_ATP01]$ more sqlnet.ora
WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY="/media/sf_Oracle/software/db-sample-schemas-main/Wallet_ATP01")))
SSL_SERVER_DN_MATCH=yes
[oracle@linux-8 Wallet_ATP01]$ export TNS_ADMIN=/media/sf_Oracle/software/db-sample-schemas-main/Wallet_ATP01
[oracle@linux-8 Wallet_ATP01]$ sqlplus admin/xxxxx@atp01_high

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Feb 10 05:02:32 2024
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Last Successful login time: Sat Feb 10 2024 04:11:47 +05:30

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.22.0.1.0

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
feky1pod2

SQL>

 
I just used atp01_high as my connect identifier as they are already defined in the tnsnames.ora file thats bundled in the zip. 

If you already have a working TNS_ADMIN (which defaults to $ORACLE_HOME/network/admin), all you have to do is to only mention the below 2 parameters in the existing sqlnet.ora file and add the connection string for the ADB connection in the tnsnames.ora file. 
WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY="/media/sf_Oracle/software/db-sample-schemas-main/Wallet_ATP01")))
SSL_SERVER_DN_MATCH=yes

Method 2: Using TLS connection.

Now let's see how to connect to the Autonomous database without the wallet files using a TLS connection. 

We can connect to the autonomous database without the wallet files as well. If we wan't to allow TLS connection to connect to Autonomous database, we need to enable Access Control List (ACL). To do that, we can click on edit adjacent to the Disabled status of Access Control List

We can configure different type of ACLs such as with specific IPs, CIDR block, VCNs or VCN OCID and the combination of these can be used in real environments. 

For our testing purpose, I'm just going to add my ip address to show the ACL and TLS connection. 

Just click on Add my IP address and click Save. This will add our IP address to the ACL and the status of the ACL will turn as Enabled. 
Now click on Edit adjacent to mTLS: Required

Uncheck the Require mTLS authentication check box to disable mTLS


Now, we have enabled TLS authentication to the Autonomous database but only the connections from my IP is allowed since we have also enabled ACLs. 

Now, as we changed the configuration, lets try to connect to the ADB without the wallet files to the database using SQL Developer first and then using SQL Plus. 

The configuration will look like the below. 

We have now selected Custom JDBC instead of Cloud wallet since we are not using TLS connection for the database. 
For the JDBC URL, the format should be 
jdbc:oracle:thin:@<connection string for the service intended> 

In my example, I'm trying to connect to atp01_high service and hence the JDBC URL will be as below. 
jdbc:oracle:thin:@(description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)
(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))
(connect_data=(service_name=g34efa58d1e9750_atp01_high.adb.oraclecloud.com))
(security=(ssl_server_dn_match=yes)))
Click on Test to see the connection status is Success. 

If we need to connect using SQL Plus, we can save this connect string in the tnsnames.ora file and connect directly. 
[oracle@linux-8 admin]$ pwd
/oracle/db/db19/network/admin
[oracle@linux-8 admin]$ ls -lrt sqlnet.ora tnsnames.ora
-rw-r-----. 1 oracle oinstall 1629 Feb  7 04:33 tnsnames.ora
-rw-r--r--. 1 oracle oinstall  619 Feb 10 05:16 sqlnet.ora
[oracle@linux-8 admin]$ rm sqlnet.ora
[oracle@linux-8 admin]$ tail tnsnames.ora
rmancat =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = linux-8.selvapc.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = rmancat.selvapc.com)
    )
  )

atp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=g34efa58d1e9751_atp01_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=yes)))
[oracle@linux-8 admin]$
[oracle@linux-8 admin]$ tnsping atp

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 13-FEB-2024 10:18:22

Copyright (c) 1997, 2023, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=g34efa58d1e9750_atp01_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=yes)))
OK (380 msec)
[oracle@linux-8 admin]$
[oracle@linux-8 admin]$ sqlplus admin/xxxxx@atp

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Feb 13 10:18:37 2024
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Last Successful login time: Mon Feb 12 2024 11:49:25 +05:30

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.22.0.1.0

SQL>
  
In the above demo, I just removed my tnsnames.ora file meaning we don't pass any location for the wallet files which is confirmed by blank o/p for Used parameter files of the tnsping. 
We are now able to connect to the autonomous database using TLS but only from my specific IP address. 
Any connection attempt from other addresses will not be allowed as per the ACL. 

Requirements of TLS connection: 

SQL Developer - version 18.2 or later
SQL Plus - Oracle instant client/database client 19.14 or later and 21.5 or later (if you need it on Windows)

References: 

Connect to Autonomous Database Using Oracle Database Tools

Happy Connecting...!!!

Sunday, January 28, 2024

ASMFD - Troubleshooting ASMCMD-9513

 I was converting my Oracle ASM disk management from udev rules to ASM Filter Driver (ASMFD) and encountered the below errors which might happen during the conversion operation and this post might help you fix the issue as quickly as possible. 


You can read more on Oracle ASMFD, configuration and migrating to ASMFD using this 19c documentation

1. Wrong/incorrect afd_diskstring value

[root@ol819-rac1 ~]#
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
No devices to be labeled.
ASMCMD-9513: ASM disk label set operation failed.
Since my disks are already pointing to /dev/oracleasm due to the udev rules, ASMFD config file picked up the afd_diskstring parameter picked up this value by default and hence the issue. Edit the /etc/oracleafd.conf config file and retry the operation

[root@ol819-rac1 ~]# cat /etc/oracleafd.conf
afd_diskstring='/dev/oracleasm/*'
[root@ol819-rac1 ~]# vi /etc/oracleafd.conf
[root@ol819-rac1 ~]# cat /etc/oracleafd.conf
afd_diskstring='/dev/sd*'
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asm-disk1 /dev/sdb1 --migrate
Invalid label string : asm-disk1
ASMCMD-9513: ASM disk label set operation failed.

[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
[root@ol819-rac1 ~]#
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
ASMDISK1                    ENABLED   /dev/sdb1
[root@ol819-rac1 ~]# ls -al /dev/oracleafd/disks
total 4
drwxrwxr-x. 2 oracle dba      60 Jan 18 10:28 .
drwxrwx---. 3 oracle dba      80 Jan 18 09:45 ..
-rw-rw-r--. 1 oracle oinstall 10 Jan 18 10:28 ASMDISK1
[root@ol819-rac1 ~]# 
Note: You can see we get the Invalid label string when provided the disk name as asm-disk1. hyphen (-) is not allowed in the disk name. If needed we can use underscore (_). I left it without any special characters. 

2. Retry label after wrong path

I used a wrong path value during label operation by providing the pointer instead of actual disk path and tried to relabel again causing ASM-9513 
We can use the kfod command to check the label details of the ASM disk. 

[root@ol819-rac1 ~]# asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
label name ASMDISK1 is already in use
ASMCMD-9513: ASM disk label set operation failed.

[root@ol819-rac1 ~]# kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                    Disk Group         Site      Failgroup           Disk-Label User     Group
================================================================================
   1:      25599 MB MEMBER    /dev/oracleasm/asm-disk1                 DATA                 #              #             ASMDISK1 oracle   dba
[root@ol819-rac1 ~]# 
In the above you can see the path is /dev/oacleasm/asm-disk1 with the Disk-label set to ASMDISK1. This might work now as I still have udev rules in place so the pointer path would work. Once I remove the udev rules, this path will not exist and would cause issues in identifying disks and my cluster won't come up as the disk contains the voting disk as well
[root@ol819-rac1 ~]# asmcmd afd_unlabel asmdisk1
disk AFD:ASMDISK1 is already provisioned for ASM
No devices to be unlabeled.
ASMCMD-9514: ASM disk label clear operation failed.

[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# afdtool -delete '/dev/oracleasm/asm-disk1'
disk /dev/oracleasm/asm-disk1 is already provisioned for ASM
No devices to be unlabeled.
[root@ol819-rac1 ~]# afdtool -delete -f '/dev/oracleasm/asm-disk1'
Unlabeled device ASMDISK1
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                    Disk Group         Site      Failgroup           Disk-Label User     Group
================================================================================
   1:      25599 MB MEMBER    /dev/oracleasm/asm-disk1                 DATA                 #              #                    # oracle   dba
[root@ol819-rac1 ~]# 
afd_unlabel fails with ASMCMD-9514 stating disk already provisioned for ASM. This means afd_label command not only labels the disk but also includes it in the list of disks managed by AFD. So AFD does not allow altering the label for a disk that is currently being managed by AFD.

We can use afdtool command to delete the label in this case. 
After unlabeling the disk, we can see the kfod command has null value for Disk-label for the disk. 

We can then use afd_label command to label the disk using the proper path as shown in this post above.

References:
 
ASMCMD AFD_LABEL --RENAME FAILS (Doc ID 2884565.1)

Wednesday, August 2, 2023

Linux Huge Pages and AUTO_ONLY option of Oracle 19c

HugePages is a feature integrated into the Linux kernel. A page is the basic unit of virtual memory, with the default page size being 4096 Bytes in the x86 architecture. Linux uses a mechanism in the CPU architecture called "Translation Lookaside Buffers" (TLB) to manage the mapping of virtual memory pages to actual physical memory addresses. So, when each time memory is accessed using the default page size, there is additional overhead as many TLB resources are consumed.


With the implementation of Huge Pages, we are grouping these small pages into a bigger chunk with consuming a single resource in TLB. Huge pages are of size 2MB (default) or 1GB (called as gigantic pages and can be configured). 

For Oracle Database, using HugePages reduces the operating system maintenance of page states and increases Translation Lookaside Buffer (TLB) hit ratio. 

Configuring Huge pages: 

There are a few conditions to be met before configuring huge pages. 

1. Linux kernel should support huge pages. 
2. Edit the memlock settings in /etc/security/limits.conf file. It is advisable to set the maximum locked memory limit to at least 90 percent of the current RAM when you enable HugePages memory. 
Make sure that you are running applications that requires Hugepages (if configured), as the locked memory can only be utilized by the application that requests them and they are not available for normal memory allocations. One advantage of using Hugepages is that the memory is pinned and will not be swapped out of memory providing better performance.

In my lab system, I have the memlock limits for oracle user set as below. Value is set in KB
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728
The current settings can be verified by ulimit -l 
[oracle@linux-8 ~]$ ulimit -l
134217728
[oracle@linux-8 ~]$
3. Calculate the number of huge pages required. 
For Oracle database requirements, we can utilize the script provided by Oracle support via support note 401749.1
Script usage is as below. Make sure all the DBs are up and running before initiating the script as the calculation is done for all shared memory segments available when the script is run, no matter it is an Oracle RDBMS shared memory segment or not. 
SQL> sho parameter sga_max_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_max_size                         big integer 4G
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
[oracle@linux-8 ~]$ ls -lrt huge*
-rwxr-xr-x. 1 oracle oinstall 3485 Aug  1 10:34 hugepages_settings.sh
[oracle@linux-8 ~]$ ./hugepages_settings.sh

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 2050
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$
We can see the Recommended setting is to set vm.nr_hugepages = 2050 pages. This can be manually calculated by us if we know what exact memory we need for the oracle database SGA and other dependent applcations. 
In our case, I run only the Oracle database with sga_max_size of 4 GB. Default hugepage size is 2MB. So, to have 4GB memory of 2 MB page size we would need 4GB / 2MB = 4096 MB / 2 MB = 2048 pages + Minimum pages (here 1 page) + 1 = 2050 pages. 

If you have multiple DBs running in the system, we need to calculate the cumulative amount for all the DBs that needs to reserve hugepages. 

What if I don't set the memlock parameter in limits.conf file and try to start up the database? 

It's interesting we sometimes set the hugepages to the correct value as suggested by the Oracle provided script but miss to set the limits.conf file to mention which user can lock how much memory. In such cases, Oracle database would fail to start with the below error indicated in the alert log. 
**********************************************************************
2023-07-31T09:43:20.619135+05:30
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

2023-07-31T09:43:20.619242+05:30
 Per process system memlock (soft) limit = 64K
2023-07-31T09:43:20.619279+05:30
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 4100M
2023-07-31T09:43:20.619346+05:30
 Available system pagesizes:
  4K, 2048K
2023-07-31T09:43:20.619412+05:30
 Supported system pagesize(s):
2023-07-31T09:43:20.619447+05:30
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2023-07-31T09:43:20.619510+05:30
     2048K                5            2050               0   ORA-27125
2023-07-31T09:43:20.619545+05:30
 Reason for not supporting certain system pagesizes:
2023-07-31T09:43:20.619603+05:30
  4K - Large pagesizes only
2023-07-31T09:43:20.619652+05:30
RECOMMENDATION:
2023-07-31T09:43:20.619688+05:30
 1. Configure system with expected number of pages for every
 supported system pagesize prior to the next instance restart operation.
2023-07-31T09:43:20.622629+05:30
 2. Increase per process memlock (soft) limit to at least 4100MB
 to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
2023-07-31T09:43:20.622708+05:30
**********************************************************************
2023-07-31T09:43:20.622896+05:30
Oradism deallocated 5 large pages of size 2097152 bytes.
2023-07-31T09:43:20.622933+05:30
SGA: Realm creation failed
2023-07-31T10:04:04.780997+05:30
Starting ORACLE instance (normal) (OS id: 38918)
2023-07-31T10:04:04.784579+05:30
************************************************************
Instance SGA_TARGET = 4096 MB and SGA_MAX_SIZE = 4096 MB
************************************************************
2023-07-31T10:04:04.784879+05:30
****************************************************
 Sys-V shared memory will be used for creating SGA
 ****************************************************
2023-07-31T10:04:04.787059+05:30
DISM started, OS id=39038
2023-07-31T10:04:08.928252+05:30
ERROR: Failed to get available system pages to allocate memory
2023-07-31T10:04:08.929435+05:30
**********************************************************************
..
..

Init parameter use_large_pages = ONLY

With Oracle database version 11.2.0.3 and above we have database parameter use_large_pages = ONLY (Default parameter value is TRUE) which would make the database start up to fail if the expected number of huge pages are not available in the system. 
SQL> sho parameter use_large_pages

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
use_large_pages                      string      TRUE
SQL> alter system set use_large_pages=ONLY scope=spfile;

System altered.

SQL>exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
[oracle@linux-8 ~]$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
[oracle@linux-8 ~]$ 
We can see Hugepages is not configured (HugePages_Total : 0). Now, let me configure hugepages less than 2050 pages and try to startup the database. 
[oracle@linux-8 ~]$ vi /etc/sysctl.conf
[oracle@linux-8 ~]$ sudo vi /etc/sysctl.conf
[sudo] password for oracle:
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ cat /etc/sysctl.conf |grep huge
vm.nr_hugepages = 2000
[oracle@linux-8 ~]$ sudo sysctl -p
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 2000
[oracle@linux-8 ~]$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    2000
HugePages_Free:     2000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4096000 kB
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Aug 1 11:22:22 2023
Version 19.10.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0

SQL> startup force
ORA-27106: system pages not available to allocate memory
Additional information: 6183
Additional information: 1
SQL>
Alert log will register the below  
Starting ORACLE instance (normal) (OS id: 70058)
2023-08-01T11:22:35.736716+05:30
************************************************************
Instance SGA_TARGET = 4096 MB and SGA_MAX_SIZE = 4096 MB
************************************************************
2023-08-01T11:22:35.736896+05:30
****************************************************
 Sys-V shared memory will be used for creating SGA
 ****************************************************
2023-08-01T11:22:35.737250+05:30
ERROR: Failed to get available system pages to allocate memory
2023-08-01T11:22:35.737286+05:30
**********************************************************************
2023-08-01T11:22:35.737319+05:30
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

2023-08-01T11:22:35.737376+05:30
 Per process system memlock (soft) limit = 128G
2023-08-01T11:22:35.737410+05:30
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 4100M
2023-08-01T11:22:35.737473+05:30
 Available system pagesizes:
  4K, 2048K
2023-08-01T11:22:35.737535+05:30
 Supported system pagesize(s):
2023-08-01T11:22:35.737572+05:30
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2023-08-01T11:22:35.737630+05:30
     2048K             2000            2050            1997   ORA-27102
2023-08-01T11:22:35.737663+05:30
 Reason for not supporting certain system pagesizes:
2023-08-01T11:22:35.737696+05:30
  4K - Large pagesizes only
2023-08-01T11:22:35.737728+05:30
RECOMMENDATION:
2023-08-01T11:22:35.737760+05:30
 1. Configure system with expected number of pages for every
 supported system pagesize prior to the next instance restart operation.
2023-08-01T11:22:35.741864+05:30
**********************************************************************
2023-08-01T11:22:35.741932+05:30
SGA: Realm creation failed
..
If I increase the huge pages to 2050 and try to start the database, the database will start allocating the expected hugepages. 

Init parameter use_large_pages = AUTO_ONLY

Starting Oracle database version 19c, we have a new option for use_large_pages = AUTO_ONLY which is the default value for 19c databases running on Exadata systems and can be configured for on premise databases as well. 

By setting the value to AUTO_ONLY, Oracle will request Linux system automatically on how many numbers of pages are required to start the database and if the server can provision the pages, the database will be started. If the request cannot be provisioned, database will fail to start. 

For demo purpose, I'll reset the hugepages to 0 and set the value use_large_pages = AUTO_ONLY using the pfile and try to start the database. 
[oracle@linux-8 ~]$ sudo vi /etc/sysctl.conf
[sudo] password for oracle:
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ sudo sysctl -p
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 0
[oracle@linux-8 ~]$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ vi /oracle/db/db19/dbs/initcdb19.ora
[oracle@linux-8 ~]$ grep large_pages /oracle/db/db19/dbs/initcdb19.ora
*.use_large_pages='AUTO_ONLY'
[oracle@linux-8 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Aug 1 11:37:59 2023
Version 19.10.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup pfile=/oracle/db/db19/dbs/initcdb19.ora
ORACLE instance started.

Total System Global Area 4294963992 bytes
Fixed Size                  9143064 bytes
Variable Size            1056964608 bytes
Database Buffers         3221225472 bytes
Redo Buffers                7630848 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    2050
HugePages_Free:        3
HugePages_Rsvd:        3
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4198400 kB
[oracle@linux-8 ~]$
You can see Oracle dynamically requested Linux to allocate 2050 pages and once the request is provisioned the database is started without us needing to configure the number of huge pages required. 

Now if I try to start another instance with 100 MB sga_max_size, if the system can provision the requested pages the database will start.  
[oracle@linux-8 ~]$ vi /oracle/db/db19/dbs/initlpdemo.ora
[oracle@linux-8 ~]$ cat /oracle/db/db19/dbs/initlpdemo.ora | grep 'sga_max\|use_large'
*.sga_max_size=100m
*.use_large_pages='AUTO_ONLY'
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ . oraenv
ORACLE_SID = [oracle] ? lpdemo
ORACLE_HOME = [/home/oracle] ? /oracle/db/db19
The Oracle base has been set to /oracle/db
[oracle@linux-8 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Aug 1 11:52:02 2023
Version 19.10.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile=/oracle/db/db19/dbs/initlpdemo.ora
ORACLE instance started.

Total System Global Area  322960856 bytes
Fixed Size                  8895960 bytes
Variable Size             239075328 bytes
Database Buffers           67108864 bytes
Redo Buffers                7880704 bytes
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
[oracle@linux-8 ~]$ 
[oracle@linux-8 ~]$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    2206
HugePages_Free:        6
HugePages_Rsvd:        6
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4517888 kB
[oracle@linux-8 ~]$
It is not always guaranteed that the requested hugepages will be provisioned by Linux. If the system cannot allocate the required hugepages, database instance creation will fail registering insufficient number of hugepages in the alert log. 
2023-08-01T11:59:33.682674+05:30
 Available system pagesizes:
  4K, 2048K
2023-08-01T11:59:33.682755+05:30
 Supported system pagesize(s):
2023-08-01T11:59:33.682796+05:30
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2023-08-01T11:59:33.682866+05:30
     2048K              151             156             151   ORA-27102
2023-08-01T11:59:33.682899+05:30
 Reason for not supporting certain system pagesizes:
2023-08-01T11:59:33.682932+05:30
  4K - Large pagesizes only
2023-08-01T11:59:33.682964+05:30
RECOMMENDATION:
2023-08-01T11:59:33.683022+05:30
 1. Configure system with expected number of pages for every
 supported system pagesize prior to the next instance restart operation.
2023-08-01T11:59:33.697313+05:30
**********************************************************************
2023-08-01T11:59:33.697997+05:30
Oradism deallocated 151 large pages of size 2097152 bytes.
2023-08-01T11:59:33.698067+05:30
SGA: Realm creation failed

Advantages and implementations

Now as we understand how use_large_pages = AUTO_ONLY works, we need to understand where this setting can be implemented and what are the advantages

1. This setting doesn't require the prior calculation of no. of hugepages to be made with the scripts. 
2. When we add additional instances and if we intend to run those instances using hugepages, no changes necessary to system parameters are needed as the calculation and allocation of huge pages is dynamic. 
3. Can be used in development or test systems where instances are added and removed frequently. 
4. Dedicated single production database (though not recommended as a change in sga_max_size will have impact on how many pages are used and locked which might cause memory starvation if over provisioned by mistake). 

Quick note: Though the allocation of hugepages is dynamic, the pages are not released after the database is shut down and is shown under Hugepages_Free. This has an impact as this memory can't be used for normal memory allocation and is a waste of resource until next reboot. 

References:

Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)


Happy Huge Paging...!!!

Saturday, June 24, 2023

Using Real time SQL Monitor in command line!

Query optimization or tuning is an interesting subject. Sometimes we would get complaints from application folks that their query is running slow in the timelines of running 4 hours against 1 or 2 hrs normal execution. During these times, we can't wait for the query to run to its completion only to see the query didn't run as expected. 

How can we find whether the query is running as expected? One of the simple ways is to run the explain plan to see whether the plan has changed from the existing one and is the join methods and filters are applied as expected and also whether the cost of the operation is reduced. But the problem here is Oracle can totally predict a wrong plan (due to missing stats for example) or with the adaptive plans can change its plan during run time as well. 

In order to know the details of the query execution we need to wait until the query completes. This is not welcome in all the situations as DBA can't wait for the query execution to complete before they decide on next course of investigation plan.

We can use DBMS_XPLAN.display_cursor or ?/rdbms/admin/utlxpls.sql or similar tools to get the explain plan of the query but for these to work to give the detailed statistics, the query should be run to its completion. Waiting for 2 or 3 hours just to check the execution details of the query seems not fun. 

Starting version 11g, Oracle introduced a feature called Real time SQL Monitoring. This is such a cool and useful feature where the stats can be seen in real time even the query is currently running. 

Requirements to use Real time SQL Monitor: 

  1. Diagnostics and Tuning Pack license.
  2. STATISTICS_LEVEL= 'TYPICAL' or 'ALL'
Methods to run Real time SQL Monitor: 
  1. Oracle Enterprise Manager
  2. SQL Developer
  3. Command line. 
Conditions where SQL is monitored by default: 
  1. When the SQL has consumed at least five seconds of the CPU or I/O time in a single execution
  2. When the SQL execute in parallel
  3. When the SQL has /*+ monitor */ hint

In this post, we are going to look into how to run Real time SQL Monitor using command line and view its results which will be useful to analyze the query execution

There are 2 packages via which sql can be monitored in real time. 

DBMS_SQLTUNE
DBMS_SQL_MONITOR

DBMS_SQLTUNE is a package that deals with tuning SQL on demand and is available from Oracle 11g onwards. The package deals with SQL profiles, creating and executing tuning tasks, tracing SQL, monitor SQL and reporting monitored SQL, etc.. We will look how to report real time SQL monitor using DBMS_SQLTUNE with an example. 

To report real time SQL, all we need is the sql_id along with the report type. By default, the report will be of text format but we have an option to get the report in nice and tidy html format. If the report needs to be generated only for a specific sql_plan_hash_value, then the value for this parameter can be provided. 

Usage example: 

DBMS_SQLTUNE

I'm now running a query with /*+monitor */ hint as this query takes less than 5 sec but I need to check the path the query has taken. 
Once the query has run or if it is a long running query we can check the v$sql_monitor view to see if the query has been captured for real time monitoring. We can then use the dbms_sqltune.report_sql_monitor procedure to report the real time statistics of the query. 
SQL> alter session set container=odb1;

Session altered.

SQL> set lines 200
SQL> select /*+ monitor */ * from oe.order_items join oe.orders on oe.order_items.order_id=oe.orders.order_id;

  ORDER_ID LINE_ITEM_ID PRODUCT_ID UNIT_PRICE   QUANTITY   ORDER_ID ORDER_DATE                     ORDER_MO CUSTOMER_ID ORDER_STATUS ORDER_TOTAL SALES_REP_ID PROMOTION_ID
---------- ------------ ---------- ---------- ---------- ---------- ------------------------------ -------- ----------- ------------ ----------- ------------ ------------
      2354            1       3106         48         61       2354 14-JUL-08 05.18.23.234567 PM   direct           104            0       46257          155
      2354            2       3114       96.8         43       2354 14-JUL-08 05.18.23.234567 PM   direct           104            0       46257          155
      2354            3       3123         79         47       2354 14-JUL-08 05.18.23.234567 PM   direct           104            0       46257          155
      2354            4       3129         41         47       2354 14-JUL-08 05.18.23.234567 PM   direct           104            0       46257          155
      2354            5       3139         21         48       2354 14-JUL-08 05.18.23.234567 PM   direct           104            0       46257          155
...
...
...
      2458            2       3123         79        112       2458 16-AUG-07 02.34.12.234359 PM   direct           101            0     78279.6          153
      2458            3       3127      488.4        114       2458 16-AUG-07 02.34.12.234359 PM   direct           101            0     78279.6          153
      2458            4       3134         17        115       2458 16-AUG-07 02.34.12.234359 PM   direct           101            0     78279.6          153

665 rows selected.

SQL>
SQL> SET LINESIZE 300
SQL> COLUMN sql_text FORMAT A100
SQL> SELECT sql_id, status, sql_text FROM v$sql_monitor where sql_text like '%monitor%';

SQL_ID        STATUS              SQL_TEXT
------------- ------------------- ----------------------------------------------------------------------------------------------------
abf24z16pukw9 DONE (ALL ROWS)     select /*+ monitor */ * from oe.order_items join oe.orders on oe.order_items.order_id=oe.orders.orde
                                  r_id

SQL> SET LONG 1000000
SQL> SET FEEDBACK OFF head off
SQL> spool monitor_sql.html
SQL> SELECT DBMS_SQLTUNE.report_sql_monitor(sql_id =>'abf24z16pukw9',type=> 'HTML') AS report FROM dual;


o/p will contain lots of html contents
as the report is generated as html.. 
...
...
...



SQL> spool off
SQL> 
We now have the report generated by name monitor_sql.html. We can view the report to see the real time execution plan statistics of the query. 


Since the query is now completed, we see the complete execution plan statistics and the completion of the query is 100%. This report is very useful when there is a long running query and if we need to investigate what's causing slowness. 

To showcase this, I just inserted 3 fold data into sales table but didn't gather statistics and ran a query without any filters. 
The first report generated shows below. 


The query runs for a very long time and the subsequent reports shows the progress of the query until it completes. Here, we can find that the query expected 919k rows from the sales table but the actual rows returned is 4M which is 4 folds higher than the expected value and hence we can easily say that the statistics is not proper for the table. 


DBMS_SQL_MONITOR

DBMS_SQL_MONITOR package provides information about Real-Time SQL Monitoring and Real-Time Database Operation Monitoring whereas DBMS_SQLTUNE is a comprehensive package consisting of all SQL related tuning capabilities and is an interface for SQL tuning on demand. 
Generating real time sql monitoring report using DBMS_SQL_MONITOR is similar to using DBMS_SQLTUNE and the report is also generated same as this. 
DBMS_SQL_MONITOR is introduced in 12c and we can now monitor SQL and PL/SQL calls containing SQL as well. 

Usage will be as below 

SQL> SET LONG 1000000
SQL> SET FEEDBACK OFF head off
SQL> spool monitor_sql.html
SQL> SELECT DBMS_SQL_MONITOR.report_sql_monitor(sql_id =>'1uhbtu4azdyqh',type=> 'HTML') AS report FROM dual;

SQL> spool off
SQL> 
Report generated will be similar to the one we saw before using DBMS_SQLTUNE with just the package name change. 


While SQL monitoring report of type => HTML gives a well formatted html report to analyze the query execution, we can generate ACTIVE report which would provide much better and additional information such as optimizer environment, parallel activity section for parallel queries, etc. Active report is my favorite go to tool for query tuning. 

So how would you generate the SQL monitor ACTIVE report? Use type => ACTIVE in the REPORT_SQL_MONITOR function as below. 
SQL> SET LONG 1000000  LONGCHUNKSIZE 1000000  LINESIZE 1000  PAGESIZE 0  TRIM ON  TRIMSPOOL ON  ECHO OFF  FEEDBACK OFF
SQL> spool monitor_sql.html
SQL> SELECT DBMS_SQLTUNE.report_sql_monitor(sql_id =>'abf24z16pukw9',type=> 'ACTIVE') AS report FROM dual;
...
...
...

SQL> spool off
The report generated will be like below. 


We can see that the report is kind of interactive where we can change the execution plan to graphical mode if one wishes to understand better and we can toggle between various options to check the complete picture of where the query is executed like the Optimizer Environment, etc. Timeline section will keep updating if we generate the report frequently while the query is running. 

When the graphical mode is toggled, we get the easily understandable view of the execution plan



Hope this helps investigate the real time sql using command line. 

References: 

DBMS_SQLTUNE (oracle.com)
DBMS_SQL_MONITOR (oracle.com)
Monitoring SQL Statements with Real-Time SQL Monitoring (Doc ID 1380492.1)

Happy Tuning...!!! 

P.S: I'm not able to generate the real time reports on the Database version 23 Free and have raised a community discussion. Let's wait to hear on this.. 

Update: 
The community discussion has been answered by SQL Developer fame Jeff Smith. 
DIAGNOSTIC+TUNING pack is by default disabled and we need to enable it to generate the reports. 

Also, I have added the section of how to generate Active SQL monitor reports. 

Friday, April 7, 2023

Oracle Transparent Application Continuity (TAC) - Part 2

I have started a transaction in my bank account and money got deducted but the screen doesn't get me the confirmation of the transaction and ended with a time out error. There could be multiple reason and one of the reason being one of the node on which the session was working crashed.

How many of you encountered this situation?

Now, if the bank/application once recovered and connected to the database (on a surviving node) and there is no details of the previous transaction available to the new session just got connected. What happens to the data? Would it get committed?

If the application tries to run the same previous transaction, money might be deducted twice. If the application doesn't retry, your transaction might not be carried out and you might miss some important last date of bill payment

So, how does Oracle handles this? 

In my last post, I explained about how Transparent Application Failover works and how it would be a great advantage for applications that mostly reads the database. 

As a continuation of that post, we will see about the Transparent Application Continuity on what is it and how does it benefit applications utilizing this feature of Oracle. 

Okay! We should know about Transaction Guard and Application Continuity before we dive into Transparent Application Continuity. Let's see them in brief to understand what they are.. 

Transaction Guard:

Transaction Guard provides a generic tool for applications to use for at-most-once execution in case of planned and unplanned outages. Applications use the logical transaction ID to determine the outcome of the last transaction open in a database session following an outage.
If you take the bank example, if the commit already happened and if application tries to replay the execution without knowing the status of the last transaction, this might lead to logical corruption of the data by committing duplicate transaction.

Benefits of Transaction Guard includes preserving the commit outcome, ensuring known outcome of every transaction and it can be used to provide at-most-once transaction execution

Let's see the illustration below



DB node crashed when: 
Session 1 committed the transaction but acknowledgement is not yet returned to application. Application now don't know status of the previous transaction
Session 2 is idle, hence no impact to the session even this session goes off
Session 3 issues the commit but before the commit is received in the database

So without transaction guard, session 1 and 3 won't be able to proceed further since they don't know the status of the previous transaction. Client is left unknown whether transaction is committed. Logical corruption is possible if application try to rerun the transaction and missing data is possible if the transaction is not retried. This is why it is difficult to failover DML when a failure occurs on a specific node as the application fails to recognize that the last submission has committed, or shall commit sometime soon, or has not run to completion. 

How do we resolve this mystery? What if we store the commit outcome in the database itself before letting the application/client know? This is what was done by introducing LTXID in Oracle database version 12.1.0.1. 

Durability of commit is saved as a Logical Transaction Identifier (LTXID) at commit for all supported transaction types and is unique for each transaction. Applications can use this LTXID to determine the commit status. LTXID is persisted on commit and reused following a rollback. The transaction history table maintains the mapping of LTXIDs to database transaction. LTXID_TRANS table is created to store the LTXIDs and is not accessible by non-DBA user. 
Hope it is clear what exactly is transaction guard provides us. 

Transaction Guard is enabled using proper database service configuration and the requirements are as follows

1. Use a non-default service
2. Set the property COMMIT_OUTCOME=TRUE in the service to determine COMMIT is accessible and to get the commit outcome of the last transaction. RETENTION_TIMEOUT can be increased if necessary (default is 24 hrs)
3. Ensure that FAN is configured with ONS to talk with 12c database clients
 
In depth understanding of Transaction Guard is provided in this whitepaper

If I had to modify the orabliss_taf service that we created in our previous post to include transaction guard, we need to modify the service as below
srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac2 bin]$ srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac2 bin]$ srvctl config service -db cdbrac -service orabliss_taf |egrep "Commit|Retention"
Commit Outcome: true
Retention: 86400 seconds
[oracle@ol819-rac2 bin]$

Connection Pools and Application Continuity: 

Application Continuity was first introduced for JDBC thin applications in Oracle 12.1 which uses an Oracle Connection pool such as Universal Connection Pool (UCP) for JDBC clients and the support for OCI was enhanced in Oracle 12.2. Connection pools can also be configured for third party applications as well. 

A connection pool is a cache of database connection objects. The objects represent physical database connections that can be used by an application to connect to a database. At run time, the application requests a connection from the pool which reuses the connection objects and reduces new connection object creation. The pool also leverages many high availability and performance features available through an Oracle RAC database. These features include Fast Connection Failover (FCF), Run-time connection Load Balancing (RLB), and Connection Affinity.

An LTXID is associated with an application request at check-out from the connection pool, and is disassociated from the application request at check-in back to the pool. Between check-out and check-in, the LTXID on the session is exclusively held by that application request meaning the LTXID is associated with the next connection session. This allows to detect duplication and failover and also basic replay by third party containers replaying the last request after a non recoverable outage. 

Application Continuity makes use of Transaction Guard (LTXID) and replays database requests exactly as they were originally submitted, to ensure a result consistent with the request. To do this, it first restores and validates the session on surviving instance and replays as necessary. 
Mutables such as NEXTVAL of sequences and calls to sysdate are taken care by using the same values when they are replayed. This is accomplished by calls to Transaction Guard. 

So KEEP privileges for mutables are required to be provided to users.

To configure Application Continuity the following parameters are needed in the service configuration. 


So for Application Continuity to be enabled in the Oracle RAC, I have to modify my service as below. 
srvctl modify service -db cdbrac -service orabliss_taf -failover_restore LEVEL1 -failoverdelay 30 -failoverretry 5 -failovertype TRANSACTION -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE
As a Database Administrator, Application Continuity has a major blocker for implementation since there needs to be code changes in the application to make use of both connection pools and get LTXID return codes and program as necessary. In this post, we are not going to test how Application Continuity works as that requires client coding. 

With Oracle 18c, Oracle introduced Transparent Application Continuity which eliminates the use of connection pools (still is recommended) and in Oracle 19c, it is further enhanced and extends support to applications that do not use connection pools. 

Transparent Application Continuity (TAC):

Transparent Application Continuity is a mode of Application Continuity that transparently tracks and records session and transactional state so that a DB session can be recovered following an outage. There are no code changes involved in the application and DBA doesn't even needs to know of application code to implement TAC. TAC uses LTXID and a state tracking infrastructure 

To use TAC, the following configuration/steps can be followed

  1. Use a supported client. SQLPLUS 19c or later is supported and we will be using this in our demo. Few other clients are Oracle JDBC Replay Driver 18c or later, UCP 18c or later, OCI Session Pool 19c or later, etc. 
  2. If using connection pool, connections should be returned to the connection pool
  3. Set FAILOVER_RESTORE to AUTO
  4. Enable mutables use in application by providing KEEP grants to Sequences and date time to users. Oracle 19c automatically KEEPs the mutables for sql. 
  5. SIDE EFFECTS (like an extenal action such as sending emails, transfer files, etc) are disabled.
With this change, if I need to enable TAC in my database I have to modify my service with below parameters (compare this with Application Continuity)



You may notice failover_restore and failovertype both defined as AUTO to enable TAC. My service modification would be as below. 
srvctl modify service -db cdbrac -service orabliss_taf -failover_restore AUTO -failoverdelay 30 -failoverretry 5 -failovertype AUTO -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE

Demo of Transparent Application Continuity (TAC): 

Let's check how TAC gives a seemless application failover with DML statements by doing the following. 
  1. Connect to database using SQLPLUS with oracle_taf without Transaction Guard enabled (without -commit_outcome=TRUE)
  2. Perform a DML and check ltxid_trans table from a different SYS session
  3. Shut down instance running DML and observe error
  4. Modify oracle_taf to enable Transaction Guard and connect to database using the service
  5. Perform DML and check ltxid_trans table from a different session
  6. Modify oracle_taf to enable Transparent Application Continuity 
  7. Connect to database using the TAC enabled service
  8. Perform DML and shut down instance running DML and observe the results
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

<<< Removed Banner >>> 

dummy@node1> select count(*) from test_table;

  COUNT(*)
----------
   8199536

dummy@node1> delete from test_table where object_id=57732;

112 rows deleted.

dummy@node1> commit;

Commit complete.

dummy@node1>

Check ltxid_trans table
[oracle@ol819-rac1 ~]$ sql / as sysdba

<<< Removed Banner >>>

SQL> @/home/oracle/ccon
Enter value for container: orabliss
old:alter session set container=&container
new:alter session set container=orabliss

Session altered.

SQL>
SQL> select * from ltxid_trans;

no rows selected

SQL> 
I'll now trigger a delete on the table and while the delete is running midway, I'll shut the OS (node 2) abruptly to create a crash scenario. 
sys@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL    SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status                                        Module
__________ ______ _________ _________ ___________ __________________ __________ _________________________ __________ ___________ _____________________________________________
         2    424      1281           DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus

sys@node1> 

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
424

dummy@node1> delete from test_table where rownum<500000; 

499999 rows deleted.

dummy@node1>

<<< Shut down node 2 abruptly to create crash scenario >>>

dummy@node1> commit;
commit
     *
ERROR at line 1:
ORA-25405: transaction status unknown


dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)              460 SELECT        BASIC      YES

dummy@node1> select count(*) from test_table;

  COUNT(*)
----------
   8199424

dummy@node1>

We can see the transaction didn't commit even though the session got failed over to the other instance. 
What if there is a select involved with the DML statement? Let's check that out..
dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
430

dummy@node1> insert into test_table select * from test_table;

 
Session 430 from node 1 got connected to node 2 and started an insert statement. While the statement is under process, I shutdown the node 2 abruptly. 
sys@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL    SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ _________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         2    430     10274           DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus

<<< Shut down node 2 here... >>>

sys@node1> @/home/oracle/sessions.sql

no rows selected

 
After sometime, when I cancelled the session I got the below error
dummy@node1> insert into test_table select * from test_table;

^Cinsert into test_table select * from test_table
*
ERROR at line 1:
ORA-25408: can not safely replay call


dummy@node1> 
This is because the fact that Transparent Application Failover (TAF) cannot handle transactions that include data manipulation language (DML). In this case, we need to log out of application, reconnect and rerun the transaction. We can see the session got failedover to instance 1

dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)              175 SELECT        BASIC      YES

dummy@node1>

We will now modify the service to enable Transaction guard and check the ltxid_trans table to check commit data details. Note: We are not going to write code to get the commit_outcome and test but will just see the contents of the ltxid_trans table. We will proceed with TAC once this check is done.
[oracle@ol819-rac1 ~]$ srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac1 ~]$ srvctl config service -db cdbrac -service orabliss_taf |egrep "Commit|Retention"
Commit Outcome: true
Retention: 86400 seconds
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
423

dummy@node1> delete from test_table where rownum<500000;

499999 rows deleted.

dummy@node1> commit;

Commit complete.

dummy@node1>
SYS@node2> select * from ltxid_trans;

   MAJ_VERSION    MIN_VERSION    INST_ID         DB_ID                        SESSION_GUID    TXN_UID    COMMIT_NO                                START_DATE    SERVICE_ID    STATE    FLAGS    REQ_FLAGS    ERROR_CODE
______________ ______________ __________ _____________ ___________________________________ __________ ____________ _________________________________________ _____________ ________ ________ ____________ _____________
             1              2          2    2960387100 F8B741BDAFB62A60E053CA38A8C0014E           111            0 07-APR-23 08.33.32.185503000 AM +05:30                1        1        0            0             0

SYS@node2>
Now, as we have enabled transaction guard once the transaction is committed, we can see the ltxid_trans table is updated with the session and transaction information. This information will be used for replaying the transaction during failover

As we have enabled only Transaction Guard as of now and if the node crashed before committing the data, we would be getting the below error message if we try to commit the data after the session failover to the surviving instance. 
dummy@node1> insert into test_table select * from test_table;

7199426 rows created.

<<< Shutdown OS abruptly >>>

dummy@node1> commit;
commit
     *
ERROR at line 1:
ORA-25402: transaction must roll back

dummy@node1> 
Now, as the final phase of our testing we will enhance the orabliss_taf to enable Transparent Application Continuity (TAC) and observe the results 

[oracle@ol819-rac1 ~]$ srvctl modify service -db cdbrac -service orabliss_taf -failover_restore AUTO -failovertype AUTO -failoverdelay 30 -failoverretry 5 -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE
[oracle@ol819-rac1 ~]$
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

...
...

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
232

dummy@node1> insert into test_table select * from test_table;

<<< Shut down node 2 while insert is running >>>

14898851 rows created.

dummy@node1> dummy@node1> commit;

Commit complete.

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
42

dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)               42 AUTO          BASIC      YES

dummy@node1>
Though I have introduced the crash in between the DML statement, the session failed over to instance 1 and completed the insert statement without any issues. 
Session state from node 1 was observed as below.
SYS@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL           SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ ________________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         1     24      7850                  DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus
         2    232     62564 19uk9dbxy0zbr    DUMMY                        3 INSERT     ol819-rac1.selvapc.com    oracle     active      SQL*Plus

SYS@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL           SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ ________________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         1     24      7850                  DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus
         1     42     53960 19uk9dbxy0zbr    DUMMY                        2 INSERT     ol819-rac1.selvapc.com    oracle     active      SQL*Plus

SYS@node1>
We can see the session 232 was running the INSERT statement against INST_ID 2 and then failed over to INST_ID 1 with SID 42. 

Conclusion: 

So the answer to my initial question "Whether DMLs will be failed over to surviving instance if one of the instance in a Oracle RAC is crashed?" is now clear. 
Oracle introduced Transaction Guard which is effectively enhanced to be utilized by Transparent Application Continuity (TAC) that can be implemented in the database without any code changes to application to have a downtime free database experience.. !

References: 



Happy TACing...!!!



Tuesday, March 21, 2023

Oracle Transparent Application Continuity (TAC) - Part 1 (TAF)

 I was asked a question a few days ago whether DMLs will be failed over to surviving instance if one of the instance in a Oracle RAC is crashed. 

I took sometime to get a proper answer to this question as I was not sure how DMLs were handled in RAC. All I know until that time was "Select" queries will be failed over to another instance. This mechanism is called Transparent Application Failover or TAF which is in Oracle from a long time since Oracle Parallel Server (OPS) introduced in Oracle 8i. 

Transparent Application Failover

When an instance fails, TAF creates a new session on the surviving node or when the SELECT mode is configured, the sessions are replayed on the surviving node. For example, let us assume we are querying a table with a million records and when the query is in midway and node fails, the session will be created on the other surviving node to the state where the query is started and the SELECT is replayed in the other node to provide output to the end user/application. This new session creation on the other node is transparent to the end user/application.

FAN and ONS

Fast Application Notification (FAN) is a critical component in all of the High Availability solutions where continuous service and continuous connections built into RAC are extended to applications and application servers. When a state of database service changes such as down or unresponsive, the new status is posted to interested subscribers via FAN events. More on FAN can be found here

Oracle Notification Services (ONS) is installed as part of installation of Oracle Grid Infrastructure or Oracle dataguard installation. ONS is responsible for propagating FAN events to all other ONS daemons it is registered with. There is one ONS daemon started per cluster node. ONS facilitates FAN and also the load balancing feature of Oracle RAC for workload management. 

Session Failover without TAF

Let's see an example of a session's behavior when we connect via the default service. This service is basic service created to connect to the database. 

My TNS entry is as below

orabliss =
    (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ol819-scan)(PORT = 1521))
        (CONNECT_DATA =
            (SERVER = DEDICATED)
            (SERVICE_NAME = orabliss.selvapc.com)
        )
    )
I have created a table TEST_TABLE under DUMMY user in the database so I'll perform the full table select and simulate the crash by turning off the node where the session is connected. 

The select statement takes ~ 50 seconds to complete and I will be terminating the node while the query is running. 

Session details: 


You can see I have connected to INST_ID 2 (ol819-rac2) from Machine ol819-rac1

First normal query execution:
SQL> select * from test_table;

=== Suppressed query output ===

440,813 rows selected.

Elapsed: 00:00:50.744
 
Now I'll be introducing a crash by turning off node 2 after firing the select query again and monitor the existing session from node 1:
...
...
...
OWNER
--------------------------------------------------------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------------------------
SUBOBJECT_NAME                                                                                                                    OBJECT_ID DATA_OBJECT_ID OBJECT_TYPE             CREATED   LAST_DDL_
-------------------------------------------------------------------------------------------------------------------------------- ---------- -------------- ----------------------- --------- ---------
TIMESTAMP           STATUS  T G S  NAMESPACE EDITION_NAME                                                                                                                     SHARING            E O A
------------------- ------- - - - ---------- -------------------------------------------------------------------------------------------------------------------------------- ------------------ - - -
DEFAULT_COLLATION                                                                                    D S CREATED_APPID CREATED_VSNID MODIFIED_APPID MODIFIED_VSNID
---------------------------------------------------------------------------------------------------- - - ------------- ------------- -------------- --------------
2019-04-17:01:42:19 VALID   N N N          1                                                                                                                                  METADATA LINK        Y N
                                                                                                     N N

ERROR:
ORA-03135: connection lost contact
Process ID: 55738
Session ID: 599 Serial number: 54452



29370 rows selected.

SQL>
Now, once the node is down the session running the select query encountered an error ORA-03135. Note the query fetched only ~29k  records of total  ~ 440k records before failure. 

Configuring TAF

TAF is configured by creating a service on RAC and assigning failover parameters as shown below. Do not use the default service that is created while creating a container database or PDB as this service will be running on all the instances and running always on a RAC system. You should not alter or modify this service. This default service cannot be disabled, relocated or restricted so it doesn't provide high availability. Instead create a specific service with unique identifier to connect to the intended database with the intended properties. 

From version 11.2 onwards we can create services using srvctl command. 

srvctl add service -db cdbrac -pdb orabliss -service orabliss_taf -tafpolicy BASIC -failovertype SELECT -clbgoal LONG -preferred cdbrac1,cdbrac2

[oracle@ol819-rac1 ~]$ srvctl add service -db cdbrac -pdb orabliss -service orabliss_taf -tafpolicy BASIC -failovertype SELECT -clbgoal LONG -preferred cdbrac1,cdbrac2
[oracle@ol819-rac1 ~]$ srvctl start service -db cdbrac -service orabliss_taf
[oracle@ol819-rac1 ~]$ srvctl status service -db cdbrac -service orabliss_taf
Service orabliss_taf is running on instance(s) cdbrac1,cdbrac2
[oracle@ol819-rac1 ~]$

We now see the details of the services from the database as below


-tafpolicy accepts only BASIC from version 11.2 meaning the failover occurs only at the time of failure and there is no pre-connect option available. 

-failovertype of SELECT enables the read on other surviving node once there is a crash on the session's existing node. Other option for TAF is SESSION where upon node failure the session is connected on the surviving instance but does not continue the SELECT.

Now, my TNS entry is as below using the newly created service

orabliss_taf =
    (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ol819-scan)(PORT = 1521))
        (CONNECT_DATA =
            (SERVER = DEDICATED)
            (SERVICE_NAME = orabliss_taf.selvapc.com)
        )
    )
I'm connecting to the DB using the orabliss_taf service and simulate the crash as before. 
SQL> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)              606 SELECT        BASIC      NO

Elapsed: 00:00:00.00
SQL> select * from test_table;

=== Suppressed query output ===

=== Introduced the crash while query is running midway ===

440813 rows selected.

Elapsed: 00:08:01.75
SQL>
SQL> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)               26 SELECT        BASIC      YES

Elapsed: 00:00:00.06
SQL>
While the original session was created with SID=606, the session failed over to node 1 and have a new SID=26. We can see the FAILED_OVER column also changed to YES. 
The query also took a significantly longer time of ~ 8 minutes against ~ 50 seconds as the failover happens to the other node and then the select has to start again from the beginning after the instance recovery is completed. So this 8 minute is a combined time of all the failover activities along with the query execution time where the session waits for the instance recovery as well.

We can see the alert log will register the below details indicating the instance recovery due to node 2 crash. This is just to check whether instance recovery is completed properly
Reconfiguration started (old inc 4, new inc 6)
List of instances (total 1) :
 1
Dead instances (total 1) :
 2
My inst 1
publish big name space -  dead or down/up instance detected, invalidate domain 0
 Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
* dead instance detected - domain 2 invalid = TRUE, need cdb-level instance recovery
* dead instance detected - domain 3 invalid = TRUE, need cdb-level instance recovery
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
...
...
 Post SMON to start 1st pass IR
2023-03-18T10:11:45.794262+05:30
CDB IR excluding pdb 2 which was cleanly closed.
2023-03-18T10:11:45.794383+05:30
Instance recovery: looking for dead threads
2023-03-18T10:11:45.812848+05:30
Beginning instance recovery of 1 threads
...
...
...
Started redo application at
 Thread 2: logseq 83, block 157204, offset 0
...
...
...
2023-03-18T10:11:46.025646+05:30
Redo thread 2 internally disabled at seq 84 (SMON)
CDB instance recovery complete: pdb 2 valid 1 (flags x10, pdb flags x40080)
CDB instance recovery complete: pdb 3 valid 1 (flags x10, pdb flags x40080)
CDB instance recovery complete: pdb 0 valid 1 (flags x10, pdb flags x80)
...
...
So, the Transparent Appliation Failover (TAF) option is great for applications which reads a lot as they don't have to worry about DB getting down as the services will get the work done transparent to the application. 

Now, what if a batch job runs in the database performing millions of DML and then the node is crashed? This could happen and mostly it's not by software error but sometimes due to patching window where applications are not appraised of the schedule, issues with underlying OS or storage causing DB node to crash, etc,. What would be the condition of the database sessions? Will the DMLs continue to run on the other surviving node? Do we have any automated mechanism to take care of this situation for us? 

Let's see about that in the next post. 

References: 


Happy TAFing...!!!