Showing posts with label Troubleshooting. Show all posts
Showing posts with label Troubleshooting. Show all posts

Wednesday, December 17, 2025

Playing with MCP Server for Oracle Database - SQLcl MCP Server & Claude Desktop client

I was trying to connect to my database using SQLcl MCP Server/protocol using Claude desktop so I can make use of Claude AI to get my work done on the database.
To know more on what is MCP protocol, MCP server/client, setting up Claude desktop on how to connect to the database, etc., I would suggest you to go through the below posts and documentation.

https://blogs.oracle.com/database/introducing-mcp-server-for-oracle-database
https://www.thatjeffsmith.com/archive/2025/07/getting-started-with-our-mcp-server-for-oracle-database/
https://docs.oracle.com/en/database/oracle/sql-developer-command-line/25.2/sqcug/using-oracle-sqlcl-mcp-server.html

There are many MCP clients we can use such as Claude desktop, VS code, Copilot, Cursor, etc. 
See https://modelcontextprotocol.io/clients for available clients at this time.

I chose Claude desktop since I already use Claude and configuration seems a bit easy at the first go. 

Make sure you have SQLcl version 25.2 and above. Else you will get the below error

2025-12-13T01:33:00.679Z [error] [sqlcl] Unexpected end of JSON input
2025-12-13T01:33:00.680Z [error] [sqlcl] Unexpected token 'B', "Bad Option: -mcp" is not valid JSON
MCP Server support is introduced only from SQLcl version 25.2.0.

My configuration file looks like below:

  {
  "mcpServers": {
    "sqlcl": {
      "command": "D:\\Oracle\\sqlcl\\bin\\sql",
      "args": ["-mcp"]
    }
   }
  }

See that we need \\ since I'm on Windows for the json escaping rules or you can simply use / in the path separator. Choice is yours.

My saved connection in SQLcl is configured as below 

SQL> conn -save cline_mcp -savepwd moviestream/oracle@192.168.56.126:1521/freepdb1
Name: cline_mcp
Connect String: 192.168.56.126:1521/freepdb1
User: moviestream
Password: ******
Connected.
SQL>
SQL> cm show cline_mcp
Name: cline_mcp
Connect String: 192.168.56.126:1521/freepdb1
User: moviestream
Password: ******
SQL>
SQL> cm test cline_mcp
Oracle AI Database 26ai Free Release 23.26.0.0.0 - Develop, Learn, and Run for Free
Connection Test Successful
SQL>

All is well until now. Then started the actual issue when I try to connect and get some details from the database. 

 
I had very limited roles/privileges (CONNECT, RESOURCE, CREATE TABLE) granted to MOVIESTREAM user so that we don't expose unnecessary data to the LLM than what is required.
You can see from the above, the client is successfully connecting to the database and we have the following as a response. 

 

Response

### DATABASE CONNECTION ESTABLISHED ###
Successfully connected to: **cline_mcp**
Response


### DATABASE ENVIRONMENT CONTEXT ###
Response

**1. Oracle Database Version:** 
23.26.0.0.0
Response

**2. Access Mode:** 
The Oracle database is currently in 'null' mode
Response

**3. NLS Parameters:** 
The current Oracle database uses the following NLS configuration:
Database character set:
Value is not valid for option json.   Remaining options are ignored.

Value is not valid for option parameter,.   Remaining options are ignored.

 Full NLS parameters: 

Value is not valid for option json.   Remaining options are ignored.

Value is not valid for option *.   Remaining options are ignored.

 This is important for correctly interpreting data formats, encodings, and locale-specific behavior.
Response

Here’s the Oracle database context you need: version, open mode, and NLS parameters. Use this information to run the following queries accurately and ensure they align with the database’s current environment and settings.
 

From the above response, Claude got the Access Mode as 'null' which is expected as the connect runs query against v$database to get the open mode of the database. User don't have privilege. 

NLS Parameters is also messed up in the response.
The client actually runs the below queries against database once the connection is successful to get the NLS related data.

SELECT * FROM nls_database_parameters
SELECT parameter, value FROM nls_session_parameters

These queries are not getting registered in the DBTOOLS$MCP_LOG table. 

DBTOOLS$MCP_LOG table keeps track of all the queries executed by the MCP client for future auditing. The table just shows a log_message entry as "Connect to MOVIESTREAM" and not the actual queries run against database. Reason being the queries are automatically executed by SQLcl client with the connect command used to connect the user session with the database. 

In this case, after the initial connection both the queries failed to execute and subsequent chat were all just error responses as shown in the Claude chat above.

The client will have its MCP logs by name mcp-server-sqlcl.log which has the following repeated messages 

2025-12-15T01:55:13.423Z [sqlcl] [info] Message from server: {"jsonrpc":"2.0","id":12,"result":{"content":[{"type":"text","text":"### DATABASE CONNECTION ESTABLISHED ###\nSuccessfully connected to: **cline_mcp**"},{"type":"text","text":"\n### DATABASE ENVIRONMENT CONTEXT ###\n"},{"type":"text","text":"**
1. Oracle Database Version:** \n23.26.0.0.0"},{"type":"text","text":"**
2. Access Mode:** \nThe Oracle database is currently in 'null' mode"},{"type":"text","text":"**
3. NLS Parameters:** 
The current Oracle database uses the following NLS configuration:
Database character set:
Value is not valid for option json.   Remaining options are ignored.
Value is not valid for option parameter,.   Remaining options are ignored.
Full NLS parameters: 
Value is not valid for option json.   Remaining options are ignored.
Value is not valid for option *.   Remaining options are ignored.
This is important for correctly interpreting data formats, encodings, and locale-specific behavior.\n"},{"type":"text","text":"Here’s the Oracle database context you need: version, open mode, and NLS parameters. Use this information to run the following queries accurately and ensure they align with the database’s current environment and settings."}],"isError":false}} { metadata: undefined }

..
..

2025-12-15T02:13:54.531Z [sqlcl] [info] Message from server: {"jsonrpc":"2.0","id":17,"result":{"content":[{"type":"text","text":"\nValue is not valid for option LLM.   Remaining options are ignored.\n"}],"isError":false}} { metadata: undefined }

..
..

2025-12-15T02:18:55.109Z [sqlcl] [info] Message from server: {"jsonrpc":"2.0","id":21,"result":{"content":[{"type":"text","text":"### DATABASE CONNECTION ESTABLISHED ###\nSuccessfully connected to: **cline_mcp**"},{"type":"text","text":"\n### DATABASE ENVIRONMENT CONTEXT ###\n"},{"type":"text","text":"**
1. Oracle Database Version:** \n23.26.0.0.0"},{"type":"text","text":"**
2. Access Mode:** \nThe Oracle database is currently in 'READ WRITE' mode"},{"type":"text","text":"**
3. NLS Parameters:** \nThe current Oracle database uses the following NLS configuration:\nDatabase character set:\nValue is not valid for option json.   Remaining options are ignored.\n\nValue is not valid for option parameter,.   Remaining options are ignored.\n\n Full NLS parameters: \n\nValue is not valid for option json.   Remaining options are ignored.\n\nValue is not valid for option *.   Remaining options are ignored.\n\n This is important for correctly interpreting data formats, encodings, and locale-specific behavior.\n"},{"type":"text","text":"Here’s the Oracle database context you need: version, open mode, and NLS parameters. Use this information to run the following queries accurately and ensure they align with the database’s current environment and settings."}],"isError":false}} { metadata: undefined }
 
I thought if these were errors due to some permission issues against internal tables/views and tried to grant the privileges and even DBA role to check if that makes a difference
grant select on sys.v_$nls_parameters to moviestream;
grant select on sys.v_$database to moviestream;
grant dba to moviestream;

Result is the same errors. These additional privileges/roles are not needed for MCP to work properly. 
Interestingly, the queries which are run against the database is being captured in the DBTOOLS$MCP_LOG table under MOVIESTREAM schema. 

Below is a screenshot of the tables content for basic understanding


The query passed to database by the Claude client is as below

SELECT /* LLM in use is claude-sonnet-4-5 */ table_name FROM user_tables;

All the queries will be added a comment with what LLM is being used when it's executed by the MCP client and the query is executed. The query execution resulted in the error below

Value is not valid for option LLM. Remaining options are ignored.

From logs what we have is the below 
2025-12-14T01:21:23.569Z [sqlcl] [info] Message from client: {"method":"tools/call","params":{"name":"run-sql","arguments":{"model":"claude-sonnet-4-5-20250929","sql":"SELECT table_name FROM user_tables"}},"jsonrpc":"2.0","id":10} { metadata: undefined }
2025-12-14T01:21:23.599Z [sqlcl] [info] Message from server: {"jsonrpc":"2.0","id":10,"result":{"content":[{"type":"text","text":"\nValue is not valid for option LLM.   Remaining options are ignored.\n"}],"isError":false}} { metadata: undefined }
 

Basically, client sends a request with a few args and MCP server (SQLcl) sends back a error response.

Doesn't make sense on why SQLcl send error response on argument (LLM) which was not passed to it. 

After a long time trying to debug the issue by different prompts from the client, executing query directly in the SQLcl prompt with/without comments, etc., none of which resolved the issue. 

I just deleted the sqlcl directory and uninstalled Java and downloaded them both again using the same way I used to do the first time and installed Java and started the SQLcl and reconfigured Claude desktop.

This time everything started to work without any issues as expected.


I'm not sure what caused the issue but with this issue, I was able to get a proper hands-on with the MCP Server, it's working and debugging methods. Hope this post will help others as well. 



References:

https://blogs.oracle.com/database/introducing-mcp-server-for-oracle-database
https://www.thatjeffsmith.com/archive/2025/07/getting-started-with-our-mcp-server-for-oracle-database/
https://docs.oracle.com/en/database/oracle/sql-developer-command-line/25.2/sqcug/using-oracle-sqlcl-mcp-server.html


 Happy playing with AI...!!! 

 

 

Sunday, January 28, 2024

ASMFD - Troubleshooting ASMCMD-9513

 I was converting my Oracle ASM disk management from udev rules to ASM Filter Driver (ASMFD) and encountered the below errors which might happen during the conversion operation and this post might help you fix the issue as quickly as possible. 


You can read more on Oracle ASMFD, configuration and migrating to ASMFD using this 19c documentation

1. Wrong/incorrect afd_diskstring value

[root@ol819-rac1 ~]#
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
No devices to be labeled.
ASMCMD-9513: ASM disk label set operation failed.
Since my disks are already pointing to /dev/oracleasm due to the udev rules, ASMFD config file picked up the afd_diskstring parameter picked up this value by default and hence the issue. Edit the /etc/oracleafd.conf config file and retry the operation

[root@ol819-rac1 ~]# cat /etc/oracleafd.conf
afd_diskstring='/dev/oracleasm/*'
[root@ol819-rac1 ~]# vi /etc/oracleafd.conf
[root@ol819-rac1 ~]# cat /etc/oracleafd.conf
afd_diskstring='/dev/sd*'
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asm-disk1 /dev/sdb1 --migrate
Invalid label string : asm-disk1
ASMCMD-9513: ASM disk label set operation failed.

[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
[root@ol819-rac1 ~]#
[root@ol819-rac1 ~]# $ORACLE_HOME/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
ASMDISK1                    ENABLED   /dev/sdb1
[root@ol819-rac1 ~]# ls -al /dev/oracleafd/disks
total 4
drwxrwxr-x. 2 oracle dba      60 Jan 18 10:28 .
drwxrwx---. 3 oracle dba      80 Jan 18 09:45 ..
-rw-rw-r--. 1 oracle oinstall 10 Jan 18 10:28 ASMDISK1
[root@ol819-rac1 ~]# 
Note: You can see we get the Invalid label string when provided the disk name as asm-disk1. hyphen (-) is not allowed in the disk name. If needed we can use underscore (_). I left it without any special characters. 

2. Retry label after wrong path

I used a wrong path value during label operation by providing the pointer instead of actual disk path and tried to relabel again causing ASM-9513 
We can use the kfod command to check the label details of the ASM disk. 

[root@ol819-rac1 ~]# asmcmd afd_label asmdisk1 /dev/sdb1 --migrate
label name ASMDISK1 is already in use
ASMCMD-9513: ASM disk label set operation failed.

[root@ol819-rac1 ~]# kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                    Disk Group         Site      Failgroup           Disk-Label User     Group
================================================================================
   1:      25599 MB MEMBER    /dev/oracleasm/asm-disk1                 DATA                 #              #             ASMDISK1 oracle   dba
[root@ol819-rac1 ~]# 
In the above you can see the path is /dev/oacleasm/asm-disk1 with the Disk-label set to ASMDISK1. This might work now as I still have udev rules in place so the pointer path would work. Once I remove the udev rules, this path will not exist and would cause issues in identifying disks and my cluster won't come up as the disk contains the voting disk as well
[root@ol819-rac1 ~]# asmcmd afd_unlabel asmdisk1
disk AFD:ASMDISK1 is already provisioned for ASM
No devices to be unlabeled.
ASMCMD-9514: ASM disk label clear operation failed.

[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# afdtool -delete '/dev/oracleasm/asm-disk1'
disk /dev/oracleasm/asm-disk1 is already provisioned for ASM
No devices to be unlabeled.
[root@ol819-rac1 ~]# afdtool -delete -f '/dev/oracleasm/asm-disk1'
Unlabeled device ASMDISK1
[root@ol819-rac1 ~]# 
[root@ol819-rac1 ~]# kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                    Disk Group         Site      Failgroup           Disk-Label User     Group
================================================================================
   1:      25599 MB MEMBER    /dev/oracleasm/asm-disk1                 DATA                 #              #                    # oracle   dba
[root@ol819-rac1 ~]# 
afd_unlabel fails with ASMCMD-9514 stating disk already provisioned for ASM. This means afd_label command not only labels the disk but also includes it in the list of disks managed by AFD. So AFD does not allow altering the label for a disk that is currently being managed by AFD.

We can use afdtool command to delete the label in this case. 
After unlabeling the disk, we can see the kfod command has null value for Disk-label for the disk. 

We can then use afd_label command to label the disk using the proper path as shown in this post above.

References:
 
ASMCMD AFD_LABEL --RENAME FAILS (Doc ID 2884565.1)

Tuesday, November 7, 2023

Oracle Enterprise Manager - undeploy deprecated plug-ins

 In my previous post, I explain on how to upgrade Oracle Enterprise Manager (OEM) from 13.4 to 13.5. In my lab environment, I have not installed any additional plug-ins. But this not the case when we did the same in live production environment. We had one plug-in which required to be migrated instead of upgraded as below. 


This is clearly indicated when we use GUI to perform the upgrade. If you are using command line to upgrade the OEM, then these prechecks are written to a file which gets buried in the huge number of lines as we tend to check only for any errors or warnings. This plug-in migration doesn't pop out as a warning or an error in the logs. 

The first attempt to upgrade resulted in OMS not starting up with incompatible plug-in version from OMS. The error matches with the symptoms indicated in support note EM 13c: Upgrade Fails At OMS startup With Error 'Plugin's Repository version is not compatible with Plugin's OMS version' (Doc ID 2491277.1)

Solution to the issue was to restore the old environment (gives you the reason why backing up the old home is very important), undeploy the deprecated plug-in on all agents and OMS and then perform the upgrade again. 

So where do we find which plug-ins are deprecated and how to undeploy them? 

You hit the following 2 support notes 

The check list note 13.5: Checklist for Upgrading Enterprise Manager Cloud Control from Version 13.3/13.4 to 13.5 (Doc ID 2761728.1) which talks about the obsolete plugins of 13.5 

The following plug-ins are obsolete and support for these plug-ins is discontinued. These plug-ins must be undeployed from the Oracle Management Agents and Oracle Management Services before proceeding with the upgrade.

  • Cloud Services Management (oracle.sysman.csm)
  • Microsoft .NET Framework (oracle.em.smdn)
  • Microsoft Active Directory (oracle.em.smad)

This note doesn't talk regarding the plug-in which was indicated in our prechecks or review page. 

The second note Oracle announces deprecation of Enterprise Manager 13c ORAchk Healthchecks Plug-in (Doc ID 2935520.1) gives exactly what we are searching for. 

So, we have the dedicated note for ORAchk Healthchecks plugin is deprecated in version 13.5. 

How do we check what other plugins are installed and whether we are good to proceed with the upgrade? 

Click on Setup => Plug-ins


This will list out all available plug-ins. 

Click on View => Only Deployed

This shows us all the deployed plug-ins in our environment. 


How to undeploy the plug-in? 

Click on the plug-in we need to undeploy => Undeploy From => Management servers... 


Since this plug-in is not installed in the Management Agents, the option is grayed out and only Management Servers is available to select. 


Input Sys Password and click Continue


Check the backup question and click Undeploy

If you haven't backed up the configuration, it can be done as below


Once we click on Undeploy, the plug-in will be undeloyed from Management server and we get the confirmation.


Progress can be seen by issuing emctl status oms -detail command


 Once completed click Close

Confirm the Deployed plug-ins 

We have successfully undeployed the deprecated plug-in and upgrade can be done without any issues. 

References: 

EM 13c: Upgrade Fails At OMS startup With Error 'Plugin's Repository version is not compatible with Plugin's OMS version' (Doc ID 2491277.1)
Undeploying Plug-Ins (oracle.com)

Happy Undeploying...!!! 

Friday, September 1, 2023

Oracle Enterprise Manager OMS OPatch upgrade mess up

 I was trying to patch my lab environment Oracle Enterprise Manager (OEM) OMS to the latest version. While doing so, we have to update the OMSPatcher and OPatch utility both to the latest version available. 

After the update to both OMSPatcher and OPatch has been done, I tried running the analyze command where I got a weird error thrown at me as below. 

[oracle@linux-8 35174911]$ omspatcher apply -analyze
java.lang.NoClassDefFoundError: oracle/oplan/sdk/intg/OPlanIntgFactory
        at oracle.opatchauto.oms.OMSPatchingSession.<clinit>(OMSPatchingSession.java:75)
        at oracle.opatchauto.oms.OPatchAuto.<clinit>(OPatchAuto.java:94)
Caused by: java.lang.ClassNotFoundException: oracle.oplan.sdk.intg.OPlanIntgFactory
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 2 more

OMSPatcher failed with error code 255
[oracle@linux-8 35174911]$ 
Now, the issue might have been caused either by omspatcher update or opatch update. The key is the first line of the error (java.lang.NoClassDefFoundError) for which there are multiple hits on support note all mentioning to restore the entire middleware home to fix this mess. 

I have followed the Oracle support note Doc ID 2646080.1 and Doc ID 2728285.1 to patch OMSPatcher and OPatch utility as a pre step and how did it mess up the software? 

So, here comes the twist. Support note EM 13.4: OMSPatcher Commands Failing With Error "java.lang.NoClassDefFoundError: oracle/oplan/sdk/intg/OPlanIntgFactory" (Doc ID 2749759.1) mentions the cause could be due to renaming the OPatch directory prior to OPatch upgrade. 

So scrolling back the screen, I found that I have moved the OPatch directory to /home/oracle since this is my practice of backing up the OPatch directory for all the database patching that we do on regular basis. So in that same thought, I used mv command to backup the OPatch directory. 

The solution to the issue is to rsync the files from old OPatch directory to new OPatch directory 
[oracle@linux-8 35174911]$ which rsync
/usr/bin/rsync
[oracle@linux-8 35174911]$ rsync -a -v --ignore-existing /home/oracle/OPatch/ /oem/oracle/middleware/OPatch/
sending incremental file list
./
opatchauto
auto/
auto/core/
auto/core/bin/
auto/core/bin/opatchauto.sh
auto/core/bin/opatchautoCopy.sh
auto/core/bin/opatchautoEnv.sh
auto/core/bin/oplan.sh
auto/core/bin/patchingWallet.sh
auto/core/modules/
auto/core/modules/com.oracle.glcm.oplancore.core_13.9.4.2.jar
auto/core/modules/com.oracle.glcm.osyscore.automation_13.9.1.0.jar
...
...
...
plugins/maven/com/oracle/oracle.glcm.oplan.core/13.9.4/oracle.glcm.oplan.core-13.9.4.pom
plugins/opatchauto/
plugins/opatchauto/modules/
plugins/opatchauto/modules/oracle.glcm.opatchauto.fmw.actions.classpath.jar
scripts/

sent 1,848,369 bytes  received 1,158 bytes  3,699,054.00 bytes/sec
total size is 5,708,468  speedup is 3.09
[oracle@linux-8 35174911]$
Once this step is done, the analyze command worked as expected. 

[oracle@linux-8 35174911]$ omspatcher apply -analyze
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.4.8.0
OUI version        : 13.9.4.0.0
Running from       : /oem/oracle/middleware
Log file location  : /oem/oracle/middleware/cfgtoollogs/omspatcher/opatch2023-09-01_01-02-52AM_1.log

OMSPatcher log file: /oem/oracle/middleware/cfgtoollogs/omspatcher/35174911/omspatcher_2023-09-01_01-02-59AM_analyze.log

Please enter OMS weblogic admin server URL(t3s://linux-8.selvapc.com:7102):>
 
Take away: 

In the patching document, the below entry is clearly presented as a note 

So it was clearly my mistake to overlook the instructions provided causing the issue. 
*** Always prepare a step by step document with clear instructions before attempting the deployment ***

Since this is my test lab, I'm ok to play with the software. 
*** Never ever attempt any deployments straight on to production database/environment even when you have an emergency situation. Have a test system ready, deploy and mimic the steps which worked perfect on the production ***
Since I have done OPatch upgrade multiple times just by replacing the OPatch directory and unzipping the patch, I have been overconfident during this step causing the mess. 
*** Never blindly do anything without looking at the instructions even though you have done it a 1000 times. There is always a chance of change or improvement ***

References: 

EM 13c: How To Upgrade Enterprise Manager 13.4 Cloud Control OMSPatcher Utility to the Latest Version (Doc ID 2646080.1)EM 13.4: OMSPatcher Commands Failing With Error "java.lang.NoClassDefFoundError: oracle/oplan/sdk/intg/OPlanIntgFactory" (Doc ID 2749759.1)


Happy OPatch upgrading...!!!

Friday, April 7, 2023

Oracle Transparent Application Continuity (TAC) - Part 2

I have started a transaction in my bank account and money got deducted but the screen doesn't get me the confirmation of the transaction and ended with a time out error. There could be multiple reason and one of the reason being one of the node on which the session was working crashed.

How many of you encountered this situation?

Now, if the bank/application once recovered and connected to the database (on a surviving node) and there is no details of the previous transaction available to the new session just got connected. What happens to the data? Would it get committed?

If the application tries to run the same previous transaction, money might be deducted twice. If the application doesn't retry, your transaction might not be carried out and you might miss some important last date of bill payment

So, how does Oracle handles this? 

In my last post, I explained about how Transparent Application Failover works and how it would be a great advantage for applications that mostly reads the database. 

As a continuation of that post, we will see about the Transparent Application Continuity on what is it and how does it benefit applications utilizing this feature of Oracle. 

Okay! We should know about Transaction Guard and Application Continuity before we dive into Transparent Application Continuity. Let's see them in brief to understand what they are.. 

Transaction Guard:

Transaction Guard provides a generic tool for applications to use for at-most-once execution in case of planned and unplanned outages. Applications use the logical transaction ID to determine the outcome of the last transaction open in a database session following an outage.
If you take the bank example, if the commit already happened and if application tries to replay the execution without knowing the status of the last transaction, this might lead to logical corruption of the data by committing duplicate transaction.

Benefits of Transaction Guard includes preserving the commit outcome, ensuring known outcome of every transaction and it can be used to provide at-most-once transaction execution

Let's see the illustration below



DB node crashed when: 
Session 1 committed the transaction but acknowledgement is not yet returned to application. Application now don't know status of the previous transaction
Session 2 is idle, hence no impact to the session even this session goes off
Session 3 issues the commit but before the commit is received in the database

So without transaction guard, session 1 and 3 won't be able to proceed further since they don't know the status of the previous transaction. Client is left unknown whether transaction is committed. Logical corruption is possible if application try to rerun the transaction and missing data is possible if the transaction is not retried. This is why it is difficult to failover DML when a failure occurs on a specific node as the application fails to recognize that the last submission has committed, or shall commit sometime soon, or has not run to completion. 

How do we resolve this mystery? What if we store the commit outcome in the database itself before letting the application/client know? This is what was done by introducing LTXID in Oracle database version 12.1.0.1. 

Durability of commit is saved as a Logical Transaction Identifier (LTXID) at commit for all supported transaction types and is unique for each transaction. Applications can use this LTXID to determine the commit status. LTXID is persisted on commit and reused following a rollback. The transaction history table maintains the mapping of LTXIDs to database transaction. LTXID_TRANS table is created to store the LTXIDs and is not accessible by non-DBA user. 
Hope it is clear what exactly is transaction guard provides us. 

Transaction Guard is enabled using proper database service configuration and the requirements are as follows

1. Use a non-default service
2. Set the property COMMIT_OUTCOME=TRUE in the service to determine COMMIT is accessible and to get the commit outcome of the last transaction. RETENTION_TIMEOUT can be increased if necessary (default is 24 hrs)
3. Ensure that FAN is configured with ONS to talk with 12c database clients
 
In depth understanding of Transaction Guard is provided in this whitepaper

If I had to modify the orabliss_taf service that we created in our previous post to include transaction guard, we need to modify the service as below
srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac2 bin]$ srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac2 bin]$ srvctl config service -db cdbrac -service orabliss_taf |egrep "Commit|Retention"
Commit Outcome: true
Retention: 86400 seconds
[oracle@ol819-rac2 bin]$

Connection Pools and Application Continuity: 

Application Continuity was first introduced for JDBC thin applications in Oracle 12.1 which uses an Oracle Connection pool such as Universal Connection Pool (UCP) for JDBC clients and the support for OCI was enhanced in Oracle 12.2. Connection pools can also be configured for third party applications as well. 

A connection pool is a cache of database connection objects. The objects represent physical database connections that can be used by an application to connect to a database. At run time, the application requests a connection from the pool which reuses the connection objects and reduces new connection object creation. The pool also leverages many high availability and performance features available through an Oracle RAC database. These features include Fast Connection Failover (FCF), Run-time connection Load Balancing (RLB), and Connection Affinity.

An LTXID is associated with an application request at check-out from the connection pool, and is disassociated from the application request at check-in back to the pool. Between check-out and check-in, the LTXID on the session is exclusively held by that application request meaning the LTXID is associated with the next connection session. This allows to detect duplication and failover and also basic replay by third party containers replaying the last request after a non recoverable outage. 

Application Continuity makes use of Transaction Guard (LTXID) and replays database requests exactly as they were originally submitted, to ensure a result consistent with the request. To do this, it first restores and validates the session on surviving instance and replays as necessary. 
Mutables such as NEXTVAL of sequences and calls to sysdate are taken care by using the same values when they are replayed. This is accomplished by calls to Transaction Guard. 

So KEEP privileges for mutables are required to be provided to users.

To configure Application Continuity the following parameters are needed in the service configuration. 


So for Application Continuity to be enabled in the Oracle RAC, I have to modify my service as below. 
srvctl modify service -db cdbrac -service orabliss_taf -failover_restore LEVEL1 -failoverdelay 30 -failoverretry 5 -failovertype TRANSACTION -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE
As a Database Administrator, Application Continuity has a major blocker for implementation since there needs to be code changes in the application to make use of both connection pools and get LTXID return codes and program as necessary. In this post, we are not going to test how Application Continuity works as that requires client coding. 

With Oracle 18c, Oracle introduced Transparent Application Continuity which eliminates the use of connection pools (still is recommended) and in Oracle 19c, it is further enhanced and extends support to applications that do not use connection pools. 

Transparent Application Continuity (TAC):

Transparent Application Continuity is a mode of Application Continuity that transparently tracks and records session and transactional state so that a DB session can be recovered following an outage. There are no code changes involved in the application and DBA doesn't even needs to know of application code to implement TAC. TAC uses LTXID and a state tracking infrastructure 

To use TAC, the following configuration/steps can be followed

  1. Use a supported client. SQLPLUS 19c or later is supported and we will be using this in our demo. Few other clients are Oracle JDBC Replay Driver 18c or later, UCP 18c or later, OCI Session Pool 19c or later, etc. 
  2. If using connection pool, connections should be returned to the connection pool
  3. Set FAILOVER_RESTORE to AUTO
  4. Enable mutables use in application by providing KEEP grants to Sequences and date time to users. Oracle 19c automatically KEEPs the mutables for sql. 
  5. SIDE EFFECTS (like an extenal action such as sending emails, transfer files, etc) are disabled.
With this change, if I need to enable TAC in my database I have to modify my service with below parameters (compare this with Application Continuity)



You may notice failover_restore and failovertype both defined as AUTO to enable TAC. My service modification would be as below. 
srvctl modify service -db cdbrac -service orabliss_taf -failover_restore AUTO -failoverdelay 30 -failoverretry 5 -failovertype AUTO -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE

Demo of Transparent Application Continuity (TAC): 

Let's check how TAC gives a seemless application failover with DML statements by doing the following. 
  1. Connect to database using SQLPLUS with oracle_taf without Transaction Guard enabled (without -commit_outcome=TRUE)
  2. Perform a DML and check ltxid_trans table from a different SYS session
  3. Shut down instance running DML and observe error
  4. Modify oracle_taf to enable Transaction Guard and connect to database using the service
  5. Perform DML and check ltxid_trans table from a different session
  6. Modify oracle_taf to enable Transparent Application Continuity 
  7. Connect to database using the TAC enabled service
  8. Perform DML and shut down instance running DML and observe the results
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

<<< Removed Banner >>> 

dummy@node1> select count(*) from test_table;

  COUNT(*)
----------
   8199536

dummy@node1> delete from test_table where object_id=57732;

112 rows deleted.

dummy@node1> commit;

Commit complete.

dummy@node1>

Check ltxid_trans table
[oracle@ol819-rac1 ~]$ sql / as sysdba

<<< Removed Banner >>>

SQL> @/home/oracle/ccon
Enter value for container: orabliss
old:alter session set container=&container
new:alter session set container=orabliss

Session altered.

SQL>
SQL> select * from ltxid_trans;

no rows selected

SQL> 
I'll now trigger a delete on the table and while the delete is running midway, I'll shut the OS (node 2) abruptly to create a crash scenario. 
sys@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL    SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status                                        Module
__________ ______ _________ _________ ___________ __________________ __________ _________________________ __________ ___________ _____________________________________________
         2    424      1281           DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus

sys@node1> 

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
424

dummy@node1> delete from test_table where rownum<500000; 

499999 rows deleted.

dummy@node1>

<<< Shut down node 2 abruptly to create crash scenario >>>

dummy@node1> commit;
commit
     *
ERROR at line 1:
ORA-25405: transaction status unknown


dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)              460 SELECT        BASIC      YES

dummy@node1> select count(*) from test_table;

  COUNT(*)
----------
   8199424

dummy@node1>

We can see the transaction didn't commit even though the session got failed over to the other instance. 
What if there is a select involved with the DML statement? Let's check that out..
dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
430

dummy@node1> insert into test_table select * from test_table;

 
Session 430 from node 1 got connected to node 2 and started an insert statement. While the statement is under process, I shutdown the node 2 abruptly. 
sys@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL    SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ _________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         2    430     10274           DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus

<<< Shut down node 2 here... >>>

sys@node1> @/home/oracle/sessions.sql

no rows selected

 
After sometime, when I cancelled the session I got the below error
dummy@node1> insert into test_table select * from test_table;

^Cinsert into test_table select * from test_table
*
ERROR at line 1:
ORA-25408: can not safely replay call


dummy@node1> 
This is because the fact that Transparent Application Failover (TAF) cannot handle transactions that include data manipulation language (DML). In this case, we need to log out of application, reconnect and rerun the transaction. We can see the session got failedover to instance 1

dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)              175 SELECT        BASIC      YES

dummy@node1>

We will now modify the service to enable Transaction guard and check the ltxid_trans table to check commit data details. Note: We are not going to write code to get the commit_outcome and test but will just see the contents of the ltxid_trans table. We will proceed with TAC once this check is done.
[oracle@ol819-rac1 ~]$ srvctl modify service -db cdbrac -service orabliss_taf -commit_outcome TRUE
[oracle@ol819-rac1 ~]$ srvctl config service -db cdbrac -service orabliss_taf |egrep "Commit|Retention"
Commit Outcome: true
Retention: 86400 seconds
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
423

dummy@node1> delete from test_table where rownum<500000;

499999 rows deleted.

dummy@node1> commit;

Commit complete.

dummy@node1>
SYS@node2> select * from ltxid_trans;

   MAJ_VERSION    MIN_VERSION    INST_ID         DB_ID                        SESSION_GUID    TXN_UID    COMMIT_NO                                START_DATE    SERVICE_ID    STATE    FLAGS    REQ_FLAGS    ERROR_CODE
______________ ______________ __________ _____________ ___________________________________ __________ ____________ _________________________________________ _____________ ________ ________ ____________ _____________
             1              2          2    2960387100 F8B741BDAFB62A60E053CA38A8C0014E           111            0 07-APR-23 08.33.32.185503000 AM +05:30                1        1        0            0             0

SYS@node2>
Now, as we have enabled transaction guard once the transaction is committed, we can see the ltxid_trans table is updated with the session and transaction information. This information will be used for replaying the transaction during failover

As we have enabled only Transaction Guard as of now and if the node crashed before committing the data, we would be getting the below error message if we try to commit the data after the session failover to the surviving instance. 
dummy@node1> insert into test_table select * from test_table;

7199426 rows created.

<<< Shutdown OS abruptly >>>

dummy@node1> commit;
commit
     *
ERROR at line 1:
ORA-25402: transaction must roll back

dummy@node1> 
Now, as the final phase of our testing we will enhance the orabliss_taf to enable Transparent Application Continuity (TAC) and observe the results 

[oracle@ol819-rac1 ~]$ srvctl modify service -db cdbrac -service orabliss_taf -failover_restore AUTO -failovertype AUTO -failoverdelay 30 -failoverretry 5 -replay_init_time 1800 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE
[oracle@ol819-rac1 ~]$
[oracle@ol819-rac1 ~]$ sqlplus dummy/dummy@orabliss_taf

...
...

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
232

dummy@node1> insert into test_table select * from test_table;

<<< Shut down node 2 while insert is running >>>

14898851 rows created.

dummy@node1> dummy@node1> commit;

Commit complete.

dummy@node1> select sys_context('userenv','sid') from dual;

SYS_CONTEXT('USERENV','SID')
--------------------------------------------------------------------------------
42

dummy@node1> select program,sid,failover_type,failover_method,failed_over from v$session where sid=sys_context('userenv', 'sid');

PROGRAM                                                 SID FAILOVER_TYPE FAILOVER_M FAI
------------------------------------------------ ---------- ------------- ---------- ---
sqlplus@ol819-rac1.selvapc.com (TNS V1-V3)               42 AUTO          BASIC      YES

dummy@node1>
Though I have introduced the crash in between the DML statement, the session failed over to instance 1 and completed the insert statement without any issues. 
Session state from node 1 was observed as below.
SYS@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL           SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ ________________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         1     24      7850                  DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus
         2    232     62564 19uk9dbxy0zbr    DUMMY                        3 INSERT     ol819-rac1.selvapc.com    oracle     active      SQL*Plus

SYS@node1> @/home/oracle/sessions.sql

   INST_ID    SID    SERIAL           SQL_ID    Username    Seconds in Wait    Command                   Machine    OS User      Status      Module
__________ ______ _________ ________________ ___________ __________________ __________ _________________________ __________ ___________ ___________
         1     24      7850                  DUMMY                                     ol819-rac1.selvapc.com    oracle     inactive    SQL*Plus
         1     42     53960 19uk9dbxy0zbr    DUMMY                        2 INSERT     ol819-rac1.selvapc.com    oracle     active      SQL*Plus

SYS@node1>
We can see the session 232 was running the INSERT statement against INST_ID 2 and then failed over to INST_ID 1 with SID 42. 

Conclusion: 

So the answer to my initial question "Whether DMLs will be failed over to surviving instance if one of the instance in a Oracle RAC is crashed?" is now clear. 
Oracle introduced Transaction Guard which is effectively enhanced to be utilized by Transparent Application Continuity (TAC) that can be implemented in the database without any code changes to application to have a downtime free database experience.. !

References: 



Happy TACing...!!!



Thursday, December 29, 2022

Troubleshooting Kerberos and Oracle CMU configuration

 In one of my previous posts, I have shared complete steps to configure Oracle Centrally Managed Users (CMU) which was completely done in my personal lab environment. Though I have encountered few hiccups, I was able to set it up without big issues. 

Now I'm trying to perform a proof of concept of the same set up in my work environment where I was stuck with several different issues and crossing them one after the other. As I was working with Oracle support and trying to iron out all issues, I learnt a few things related to AD, LDAP, etc.,.


In this post, I'll share the troubleshooting steps for the issues encountered so that it could help fellow DBAs to set up in their environment. 

The following support note will be of much help but its very limited in identifying the exact issue. We can use this note as starting point to dig deeper. 

Kerberos Troubleshooting Guide (Doc ID 185897.1)

Ok, let's start with the different troubleshooting steps. This is not step by step approach and any step can be used any time as per need basis

1. Tracing:

We might just get random errors like ORA-12638, ORA-12631, etc., but it lead you to nowhere. So need to trace the session in order to work with Oracle support and provide them with what they are looking for. So how do you trace the session? 

We need to set the below parameters in sqlnet.ora in both database server (if server trace is needed) and client machine (usually this will be mandatory to have). 

TRACE_UNIQUE_SERVER = ON
TRACE_LEVEL_SERVER = 16
TRACE_DIRECTORY_SERVER = /oracle/ABC/19.0.0/network/log
TRACE_FILE_SERVER = Server_Trace
TRACE_TIMESTAMP_SERVER = ON
DIAG_ADR_ENABLED=OFF
TRACE_UNIQUE_CLIENT = ON
TRACE_LEVEL_CLIENT = 16
TRACE_DIRECTORY_CLIENT = /oracle/ABC/19.0.0/network/log
TRACE_FILE_CLIENT = Client_Trace
TRACE_TIMESTAMP_CLIENT = ON
DIAG_ADR_ENABLED=OFF
Once tracing is completed, we can switch the parameter TRACE_LEVEL_SERVER or TRACE_LEVEL_CLIENT to OFF which will stop tracing of sessions. Make sure you just turn on and turn off quickly once you test your connection as this is instance wide and all the sessions will be traced filling up your log location/disk quickly. 

You can identify the trace file by looking into the sqlnet.log file, just giving a sample log content from the sqlnet.log
***********************************************************************
Fatal NI connect error 12631, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xxxxx.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ABC)(CID=(PROGRAM=sqlplus)(HOST=xxxxx)(USER=oracle))))

  VERSION INFORMATION:
        TNS for Linux: Version 19.0.0.0.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
  Version 19.16.0.0.0
  Time: 23-DEC-2022 19:34:37
  Tracing to file: /oracle/ABC/19.0.0/network/log/client_trace_3655062.trc
  Tns error struct:
    ns main err code: 12631
    TNS-12631: Username retrieval failed
    ns secondary err code: 0
	
***********************************************************************
The logfile is indicated for the error that we have received. We can provide this log file to Oracle support for further review

2. Generating krb5 trace:

Below commands would help generate krb5 trace where we can trace actions related to okinit 

a) Set krb5_trace parameter

export KRB5_TRACE=<path>/krb5trace.trc
echo $KRB5_TRACE

b) On the database server run

$ okdstry
$ okinit <xxx>
$ oklist
$ sqlplus /@<connect_identifier>

This will generate krb5trace.trc file which can be uploaded to support for their analysis

3. Use of same case for principal name:

While creating SPN for database and generating keytab file, make sure you are using small case for creation and configuration 

I have used 
ktpass -princ ORACLE/linux-8.selvapc.com@ORABLISS.com -pass Qwerty123# -mapuser oserv@orabliss.com -crypto ALL -ptype KRB5_NT_PRINCIPAL -out Z:\linux-8-database.keytab

This resulted in error in my PoC environment. 

Instead use small case for database SPN
ktpass -princ oracle/linux-8.selvapc.com@ORABLISS.com -pass Qwerty123# -mapuser oserv@orabliss.com -crypto ALL -ptype KRB5_NT_PRINCIPAL -out Z:\linux-8-database.keytab  

sqlnet.ora should also have principal name in small case. 

SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=oracle

4. ldapbind:

The ldapbind command-line tool enables you to see whether you can authenticate a client to a server. When we set up CMU and if we are getting error, we can check whether the client machine is getting authenticated with the server by using the command as below.

ldapbind -h <AD Host> -p 389 -D <value of ORACLE.SECURITY.DN> -w <value of ORACLE.SECURITY.PASSWORD>
ldapbind -h <AD Host> -p 389 -D <value of ORACLE.SECURITY.USERNAME> -w <value of ORACLE.SECURITY.PASSWORD>
ldapbind -h <AD Host> -p 636 -U 2 -W "file:<wallet location>" -P <wallet password> -D <value of ORACLE.SECURITY.DN> -w <value of ORACLE.SECURITY.PASSWORD>
ldapbind -h <AD Host> -p 636 -U 2 -W "file:<wallet location>" -P <wallet password> -D <value of ORACLE.SECURITY.USERNAME> -w <value of ORACLE.SECURITY.PASSWORD>

The first 2 test will test authentication with TCP and the next 2 will test authentication with SSL which we would have already set up in CMU. 

Example given below: 
$ ldapbind -h xxx.xxx.example.com -p 389 -D "CN=SV-oraservice,OU=Service,OU=Accounts,DC=xxx,DC=xxx,DC=example,DC=com" -w '%*9&nnh!!GFulh!G'
bind successful
$ ldapbind -h xxx.xxx.example.com -p 389 -D SV-oraservice -w '%*9&nnh!!GFulh!G'
bind successful
$ ldapbind -h xxx.xxx.example.com -p 636 -U 2 -W "file:/oracle/grid/19.0.0/network/admin/cmu_wallet" -P W9&nnhYT# -D "CN=SV-oraservice,OU=Service,OU=Accounts,DC=xxx,DC=xxx,DC=example,DC=com" -w '%*9&nnh!!GFulh!G'
bind successful
$ ldapbind -h xxx.xxx.example.com -p 636 -U 2 -W "file:/oracle/grid/19.0.0/network/admin/cmu_wallet" -P W9&nnhYT# -D SV-oraservice -w '%*9&nnh!!GFulh!G'
bind successful
$
If there is any issue with the connectivity, we will be getting a different error. For eg, let me try with a wrong password for the service account..
$ ldapbind -h xxx.xxx.example.com -p 389 -D "CN=SV-oraservice,OU=Service,OU=Accounts,DC=xxx,DC=xxx,DC=example,DC=com" -w 'wrongpass'
ldap_bind: Invalid credentials
ldap_bind: additional info: 80090308: LdapErr: DSID-0C090439, comment: AcceptSecurityContext error, data 52e, v4563
$
You can see when I provided a wrong password, I got the Invalid credentials error as expected. 

5. LDAP force sync

When there is change in the AD side such as password change for the service account, Oracle sometimes won't sync up with the changes and will throw error. We need to force sync Oracle with the LDAP services using the below parameters. 
Note: This is needed only when you doubt that Oracle is not synced up with AD/LDAP 
SQL> alter system set "_ldap_config_force_sync_up"=TRUE;

System altered.

SQL> alter system set "_ldap_config_force_sync_up"=FALSE;

System altered.

SQL> exit

$ sqlplus /@<connect string> 
 
We just need to set _ldap_config_force_sync_up TRUE and FALSE and retry the connection to the database. 

6. Viewing available certificates in AD server

Usually certificates are provided by issuing authority or if it's internal as in this case, AD team will provide you with the certificates. In my case, if I need to check what are the certificates I might need from AD team for the configuration to work, we can check using the openssl -showcert command. 

The command is as follows 
openssl s_client -connect <AD host>:636 -showcerts
$ orapki wallet display -wallet .
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Oracle Secret Store entries:
ORACLE.SECURITY.DN
ORACLE.SECURITY.PASSWORD
ORACLE.SECURITY.USERNAME
Trusted Certificates:
Subject: CN=EXAMPLE MSPKI XXX Root CA
$
$ openssl s_client -connect abcdef12.xxx.xxx.example.com:636 -showcerts
CONNECTED(00000003)
depth=1 DC = com, DC = example, DC = xxx, DC = xxx, CN = qasmime95
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = ABCDEF12.xxx.xxx.example.com
verify return:1
---
Certificate chain
 0 s:CN = ABCDEF12.xxx.xxx.example.com
   i:DC = com, DC = example, DC = xxx, DC = xxx, CN = qasmime95
-----BEGIN CERTIFICATE-----
MIIFuDCCBKCgAwIBAgITbwAABIUbCx91qS38VQAAAAAEhTANBgkqhkiG9w0BAQsF
ADB2MRMwEQYKCZImiZPyLGQBGRYDY29tMRkwFwYKCZImiZPyLGQBGRYJaG9uZXl3
...
...
...
ZA8ptBPzUXSgSNI+5wfh3F58Tfoi8Ig07ryyR4HD3WKGDE2DDSmMdnZ8oB2cBnuJ
T1RQK8+16GKI1bQFoNRfbwZZXmOjlIJiESZvUg==
-----END CERTIFICATE-----
 1 s:DC = com, DC = example, DC = xxx, DC = xxx, CN = qasmime95
   i:CN = EXAMPLE QMSPKI XXX Root CA
-----BEGIN CERTIFICATE-----
MIIGJDCCBAygAwIBAgITfQAAAAvG9D//4L986AAAAAAACzANBgkqhkiG9w0BAQsF
ADAsMSowKAYDVQQDEyFIb25leXdlbGwgUU1TUEtJIEludGVybmFsIFJvb3QgQ0Ew
...
...
...
ZNaJmBGIBZsLM6IrUw/WGAdNdPQUbcXMBdlSGY5DaGYMGNjnoOqPHQ==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = ABCDEF12.xxx.xxx.example.com

issuer=DC = com, DC = example, DC = xxx, DC = xxx, CN = qasmime95
...
...
 
$
This check is required if you suspect that the provider team has given a prod certificate in place of non prod certificate or vice versa. You can see from above that the wallet has MSPKI XXX Root CA whereas the AD server has QMSPKI XXX Root CA. So we have an incorrect certificate in the wallet and hence it needs to be removed and proper certificate needs to be imported. 
Note: O/p of the command has been trimmed to include only needed content. 

7. Check your account availability

I know this can be an odd situation as to check our account availability but its better to check whether the account that you are trying to access is created/available in the AD and is binded to the service account. We can use ldapsearch command to check for the account existence with all the account details 

Command is as below 
ldapsearch -R -h <AD host> -p 389 -b "DC=xxxx,DC=xx,DC=com" -D <ad super user> -w "<ad user password>" "sAMAccountName=<AD user>"
$ ldapsearch -R -h abcdef12.xxxxx.xx.example.com -p 389 -b "DC=xxxxx,DC=xx,DC=example,DC=com" -D SV-oraservice -w 'password' "sAMAccountName=h368225"
CN=Selvakumar Nagulan,OU=Accounts,OU=Tier1,OU=Admin,DC=xxxxx,DC=xx,DC=example,DC=com
objectClass=top
objectClass=person
objectClass=organizationalPerson
objectClass=user
cn=Selvakumar Nagulan
...
...
 
$
Note that we use our service account as super user to check whether our DB user is properly bind to super user or not. You will get a blank response if the user is not bind or not configured. 



So that's all for now as I was able to successfully configure CMU after working with Oracle support with 2 different support tickets and internal AD team spanning over a month. You might connect with me on any issues regarding setting up CMU in your organization and I would be happy to help with my experiences troubleshooting the same. 

References: 

Oracle Centrally Managed Users (CMU) configuration

Happy Troubleshooting!!! 
 
 
 
 
 
 
 
 

Monday, December 12, 2022

Performance slowness due to Spatial component

 We had a complaint from application team that their database is running slow after the quarterly patching. We had patched many databases and only one database is complaining on slowness. The environment is Exacc and the database is a container (running version 19.16) which has multiple PDBs. So patching couldn't be a reason is the initial thought. So we stepped into investigate what is causing the slowness. 

As a first step, we pulled an AWR report for the time frame where they did some DB operations. 



We can see from above that most of the waits are related to Cluster and topping them is gc cr multi block mixedgc cr block 2-way and gc buffer busy acquire

So this means some query is running very frequently accessing 1 or 2 table blocks continuously. We need to figure out which query is running, accessing what table/index and how frequent and what's the elapsed time of the query. These information will get us to move forward. 

Since the waits are related to Cluster, let's see SQL ordered by Cluster Wait Time (Global) section under SQL Statistics



We see 2 queries that contribute to the most cluster waits. Both these are related to spatial component as the queries are as below and they had ~ 53k executions in the report
SELECT /* chk_journal_reg */ rowid FROM "MDSYS"."SDO_TXN_JOURNAL_REG" WHERE sid = :1 AND sdo_txn_idx_id = :2 AND indexptniden = :3 AND operation = :4 AND rid = :5

SELECT /* sel_dist_upd */ ROWID, ....., ins_no, del_no FROM "MDSYS"."SDO_DIST_METADATA_TABLE" WHERE sdo_index_owner = :own AND sdo_index_name = :nam AND sdo_partition_name = :ptn FOR UPDATE

Ok, these 2 queries look simple and not complicated, so what's causing this buffer busy issues? Investigation went in following direction

Check size of tables: They are too small. 

Check the execution plan of the first code with high cluster waits gave us the clue. Though we have primary key defined on all five columns (SID, SDO_TXN_IDX_ID, INDEXPTNIDEN, OPERATION, RID), the query took the path of INDEX STORAGE FAST FULL SCAN which would traverse through all the blocks unnecessarily. The query took around 4 second per execution. 

I hint the query to use the index already available (/*+ index(SDO_TXN_JOURNAL_REG SYS_C0014253) */) - this resulted in a much better plan and execution time. Execution time reduced to milliseconds. Remember this query executed around 53k times which would make a big difference in application performance. While checking for any plan change, I found that the index path was used until the patch was applied and the plan flipped

SQL> select distinct sql_id, plan_hash_value, id, operation, options
  2* from dba_hist_sql_plan where sql_id='fw4q2s9j3bckt' order by plan_hash_value, id;

          SQL_ID    PLAN_HASH_VALUE    ID           OPERATION                   OPTIONS
________________ __________________ _____ ___________________ _________________________
fw4q2s9j3bckt            1765869626     0 SELECT STATEMENT
fw4q2s9j3bckt            1765869626     1 INDEX               STORAGE FAST FULL SCAN
fw4q2s9j3bckt            2134037127     0 SELECT STATEMENT
fw4q2s9j3bckt            2134037127     1 INDEX               UNIQUE SCAN

SQL> 
So, we decided to create sql profile so that the plan stick to plan_hash_value 2134037127 using note Encouraging CBO to Pickup a Better Execution Plan Using the COE XFR SQL Profile Script (Doc ID 1955195.1)

In the mean time, Oracle support note (After Applying the 19.16 DBRU, Scheduler Jobs on Spatial Tables Take a Long Time to Execute (Doc ID 2907800.1)) mentioned a bug related to this same table and query. 

Reason for slowness is 

Running a MV refresh using the DBMS_SCHEDULER package causes spatial to use the SDO_TXN_JOURNAL_REG transaction journal table, instead of the SDO_TXN_JOURNAL_GTT transaction journal table.

So, now the solution is to revert the sql profile created and apply spatial bundle bug fix Patch 34725493 or to apply RU 19.17

Since we already had applied another one off patch for a previous issue with Spatial, we couldn't apply this bug fix patch as it was conflicting with the one off patch. Raised an SR with Oracle to get the merge patch for both the conflicting patches but Oracle couldn't provide any commitment on when the merge patch will be ready.

As we need to get the production database back to normal as soon as we can, we had to roll back the entire 19.16 RU to bring it back to pre patch state (RU 19.15) and then the application verified all the jobs are running without issues. We could also not see any cluster related wait post the patch rollback. We rolled back as 19.17 is yet to be tested in our lower environments before we promote to production.

Interestingly, Oracle maintains a note on all the patches that needs to be applied post RU patch apply under What Is The Latest Spatial Patch Bundle for 12c and Higher Databases? (Doc ID 2514624.1) So every time we apply RU to the database, we need to refer this note and apply the patch that is mentioned for the related RU as well. 

This investigation is not only a trouble shooting experience but also a knowledge gathering experience with Oracle Spatial component regarding the bugs related to Spatial and separate Spatial patch bundle, etc.,. 


Happy Trouble shooting and Patching...!!!


Saturday, May 7, 2022

Datapump & system privileges with property value = 1

Many automation tools like Autoupgrade (a sample demo is provided in this link), Zero Downtime Migration (ZDM), etc has been released by Oracle to make DBA life easier. Though we have these tools, we always run towards the most preferred and simple method of Datapump whether to jump the DBs from one server to another or one version to another if it can be done within the allowed downtime. It is such a powerful utility used by almost every DBA in their work life.

In one of my previous post, I have explained regarding few tips to make datapump job perform better. They still stand true for current version of Oracle as well with many advancements included.


Today we see regarding the privileges with property value =1 and its effect on datapump. 
In my recent migration of DB from version 11.2.0.4 to 19c, post migration using datapump, application complained stating few privileges are missing in the 19c DB. This can't be possible as I have taken a completed schema export and imported in the target database without any errors reported in the impdp logfile.
Upon investigating, ADMINISTER RESOURCE MANAGER privilege has not granted in the 19c DB though there are no errors in both expdp and impdp logfile. 
Taking a look at this Doc ID 1163383.1, a few privileges are not granted normally via grant statements but they have to be granted through specific plsql packages. 

How do we find those privileges? - The below query can provide and the result differs w.r.to DB version. Example from my source system 11g. 
SQL> select name from system_privilege_map where property=1 order by name;

NAME
----------------------------------------
ADMINISTER RESOURCE MANAGER
ALTER ANY EVALUATION CONTEXT
ALTER ANY RULE
ALTER ANY RULE SET
CREATE ANY EVALUATION CONTEXT
CREATE ANY RULE
CREATE ANY RULE SET
CREATE EVALUATION CONTEXT
CREATE RULE
CREATE RULE SET
DEQUEUE ANY QUEUE

NAME
----------------------------------------
DROP ANY EVALUATION CONTEXT
DROP ANY RULE
DROP ANY RULE SET
ENQUEUE ANY QUEUE
EXECUTE ANY EVALUATION CONTEXT
EXECUTE ANY RULE
EXECUTE ANY RULE SET
MANAGE ANY FILE GROUP
MANAGE ANY QUEUE
MANAGE FILE GROUP
READ ANY FILE GROUP

22 rows selected.

SQL>
All the above are granted via specific packages or via explicit grants on the system. 

How do we find which privileges does our source schema has from the above list which needs to be granted explicitly via packages? - The below query is an example
SQL> select p.grantee, m.name from system_privilege_map m, dba_sys_privs p
  2  where m.name=p.privilege and m.property=1 and p.grantee in ('DUMMY')
  3  order by p.grantee, m.name;

GRANTEE                        NAME
------------------------------ ----------------------------------------
DUMMY                          ADMINISTER RESOURCE MANAGER
DUMMY                          READ ANY FILE GROUP

SQL>
I have got 2 of such privileges that needs explicit grants on the target DB. 

Now, on the 19c DB we need to grant the privileges as below. A few would require explicit grant (like the READ ANY FILE GOUP) and  few would require grants via packages as below (like the ADMINISTER RESOURCE MANAGER). 
SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


13 rows selected.

SQL> BEGIN
  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SYSTEM_PRIVILEGE(
   GRANTEE_NAME   => 'DUMMY',
   PRIVILEGE_NAME => 'ADMINISTER_RESOURCE_MANAGER',
   ADMIN_OPTION   => FALSE);
END;
/  2    3    4    5    6    7

PL/SQL procedure successfully completed.

SQL> grant READ ANY FILE GROUP to DUMMY;

Grant succeeded.

SQL> @check_user_privs.sql

GRANTEE                        TYP PRIVILEGE OR ROLE
------------------------------ --- ---------------------------------------------------------------------------
DUMMY                          PRV ADMINISTER RESOURCE MANAGER
DUMMY                              ALTER SESSION
DUMMY                              CREATE ANY DIRECTORY
DUMMY                              CREATE DATABASE LINK
DUMMY                              CREATE JOB
DUMMY                              CREATE PROCEDURE
DUMMY                              CREATE SEQUENCE
DUMMY                              CREATE SESSION
DUMMY                              CREATE SYNONYM
DUMMY                              CREATE TABLE
DUMMY                              CREATE TYPE
DUMMY                              CREATE VIEW
DUMMY                              READ ANY FILE GROUP
DUMMY                              SELECT ANY DICTIONARY
DUMMY                              UNLIMITED TABLESPACE


15 rows selected.

SQL>
Now, we are all set. So while performing export and import we need to make sure to take care of these privilege grants without fail for the application to run without any issues. 

Note: Similarly, grants on sys owned objects will also be not transferred via datapump to target database. They have to be explicitly granted on the target database. 

Query used: 

check_user_privs.sql
select grantee, 'PRV' type, privilege pv 
from dba_sys_privs where grantee = 'DUMMY' union
select username grantee, '---' type, 'empty user ---' pv from dba_users 
where not username in (select distinct grantee from dba_role_privs) and
not username in (select distinct grantee from dba_sys_privs) and 
not username in (select distinct grantee from dba_tab_privs) and username like 'DUMMY'
group by username
order by grantee, type, pv;

References: 
Primary Note For Privileges And Roles (Doc ID 1347470.1)