Looking for something? Try here..

Monday, September 27, 2021

Creating Auto-Login Keystores for TDE

If we need to create a TDE column or a tablespace, first requirement is to create a keystore to save the encryption key (secret). 


Provide the location (ENCRYPTION_WALLET_LOCATION) where the wallet or keystore will be stored in sqlnet.ora. This can also be provided as database parameter WALLET_ROOT if you don't want to use the sqlnet.ora method.

[oracle@linux75-2 ~]$ cat /u01/orcl/122/network/admin/sqlnet.ora
# sqlnet.ora Network Configuration File: /u01/orcl/122/network/admin/sqlnet.ora
# Generated by Oracle configuration tools.

NAMES.DIRECTORY_PATH= (TNSNAMES)

ENCRYPTION_WALLET_LOCATION =
  (SOURCE =(METHOD = FILE)(METHOD_DATA =
    (DIRECTORY = /u01/orcl/122/network/wallet/)))
SQLNET.EXPIRE_TIME=2
[oracle@linux75-2 ~]$ 

Create a keystore 

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/u01/orcl/122/network/wallet/' IDENTIFIED BY oracle;

keystore altered.

SQL> 
Now, keystore ewallet.p12 file will be created in the location mentioned. NEVER delete, lose or tamper this file. 
We can open and close the keystore as needed. The issue is that every time we bounce the database, we have to explicitly open the keystore. 
SQL> select * from v$encryption_wallet;

WRL_TYPE             WRL_PARAMETER                                      STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- -------------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE                 /u01/orcl/122/network/wallet/                      CLOSED                         UNKNOWN              SINGLE    NO                 0

SQL> administer key management set keystore open identified by oracle;

keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE             WRL_PARAMETER                                      STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- -------------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE                 /u01/orcl/122/network/wallet/                      OPEN                           PASSWORD             SINGLE    NO                 0

SQL> exit
When I tried to upgrade the database to 19c from the current 12.2 version using Autoupgrade utility, the analyze phase failed due to non availability of Auto-login keystore even though the keystore is open when the analyze phase is initiated. 

So how do we create an auto login keystore? 
Auto-login keystore is nothing but we don't have to manually open the keystore after a restart of the database. The creation is simple by the below command. 
SQL> ADMINISTER KEY MANAGEMENT CREATE LOCAL AUTO_LOGIN KEYSTORE FROM KEYSTORE '/u01/orcl/122/network/wallet/' IDENTIFIED BY oracle;

keystore altered.

SQL> !
[oracle@linux75-2 ~]$ ls -lrt /u01/orcl/122/network/wallet/
total 12
-rw-------. 1 oracle oinstall 2408 Jun 12 23:41 ewallet_2021061218110265.p12
-rw-------. 1 oracle oinstall 3848 Jun 12 23:41 ewallet.p12
-rw-------. 1 oracle oinstall 3901 Sep 26 00:21 cwallet.sso
[oracle@linux75-2 ~]$ exit
exit
The LOCAL keyword specifies the keystore can only be opened from the local machine. 
You can see a new file by name cwallet.sso is created which is the auto login keystore file. NEVER delete, lose or tamper this file as this will be an irreversible damage to the database. 

Once the auto login keystore is created, the password keystore will no longer work and the autologin keystore will automatically precedence and login and open the keystore from the next restart. 

SQL> select * from v$encryption_wallet;

WRL_TYPE   WRL_PARAMETER                                      STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
---------- -------------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE       /u01/orcl/122/network/wallet/                      OPEN                           PASSWORD             SINGLE    NO                 0

SQL> administer key management set keystore close;
administer key management set keystore close
*
ERROR at line 1:
ORA-28389: cannot close auto login wallet


SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8628936 bytes
Variable Size             595592504 bytes
Database Buffers          461373440 bytes
Redo Buffers                8146944 bytes
Database mounted.
Database opened.
SQL> set lines 200
SQL> col wrl_parameter for a50
SQL> select * from v$encryption_wallet;

WRL_TYPE   WRL_PARAMETER                                      STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
---------- -------------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE       /u01/orcl/122/network/wallet/                      OPEN                           LOCAL_AUTOLOGIN      SINGLE    NO                 0

SQL> exit
You can see that the WALLET_TYPE is changed from PASSWORD to LOCAL_AUTOLOGIN and is also the STATUS is OPEN upon the restart of the database. 

Now, my analyze phase of the Autoupgrade also completed successfully without any errors. 


Happy Keystoring...!!!

Monday, September 6, 2021

Patching Oracle home with limited storage

 Today, while trying to apply patch on one of my test database, I encountered the following error. 

[oracle@linux75-2 32916808]$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.26
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/orcl/122
Central Inventory : /u01/oraInventory
   from           : /u01/orcl/122/oraInst.loc
OPatch version    : 12.2.0.1.26
OUI version       : 12.2.0.1.4
Log file location : /u01/orcl/122/cfgtoollogs/opatch/opatch2021-09-05_23-04-44PM_1.log

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(3702.527MB) is not available.
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file location: /u01/orcl/122/cfgtoollogs/opatch/opatch2021-09-05_23-04-44PM_1.log

OPatch failed with error code 73
[oracle@linux75-2 32916808]$ 
/u01 file system has only around 2.6 GB of free space where as the patch is looking for around 3.7 GB of free space. As this being my test machine, I don't want to add space and was looking to work around the situation. 
[oracle@linux75-2 ~]$ df -h /u01 /u02 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        10G  7.5G  2.6G  75% /u01
/dev/sda5        17G  9.3G  7.7G  55% /u02
[oracle@linux75-2 ~]$ 

Now, the downloaded patch (Patch 32916808: DATABASE JUL 2021 RELEASE UPDATE 12.2.0.1.210720) is < 1 GB in size. So why does the patch requires > 3.5 GB of free space. 

Starting with 10.2, Opatch does not backup only the affected modules, it also takes a backup of the complete affected libraries to $ORACLE_HOME/.patch_storage/<patchid>/backup/<lib_directory_name>/<library_name>

So the 3.5 GB of space is needed just to backup the existing software files so that Oracle can safely rollback if we encounter any issue during the patching process. 

So how do we over come this space issue now? 

Firstly, we need to bypass the opatch utility from performing the free space check during the patching operation 
Second, we need to figure out an alternate path to store the backup files for the duration of the patching

By running the $ORACLE_HOME/OPatch/opatch prereq -help, we can find all the prerequisites the OPatch utility is performing of which CheckSystemSpace is one of the prerequisite. 
If we start the opatch utility using -verbose option, we can see all the associated parameters taken during the execution of the opatch operation as below.. 

[oracle@linux75-2 ]$ $ORACLE_HOME/OPatch/opatch apply -verbose
Oracle Interim Patch Installer version 12.2.0.1.26
Copyright (c) 2021, Oracle Corporation.  All rights reserved.

Environment:
   OPatch.ORACLE_HOME=/u01/orcl/122
   oracle.installer.invPtrLoc=/u01/orcl/122/oraInst.loc
   oracle.installer.oui_loc=/u01/orcl/122/oui
   oracle.installer.library_loc=/u01/orcl/122/oui/lib/linux64
   oracle.installer.startup_location=/u01/orcl/122/oui
   OPatch.PLATFORM_ID=
   os.name=Linux
   OPatch.NO_FUSER=
   OPatch.SKIP_VERIFY=null
   OPatch.SKIP_VERIFY_SPACE=null
   oracle.installer.clusterEnabled=false
   TRACING.ENABLED=null
   TRACING.LEVEL=null
   OPatch.DEBUG=false
   OPATCH_VERSION=12.2.0.1.26
   Bundled OPatch Property File=properties
   ...
   ...
   ...
We can see OPatch.SKIP_VERIFY_SPACE is set to null by default. We will set this parameter to TRUE to skip space verification. 

Next, we will create a symbolic link for .patch_storage under ORACLE_HOME to a directory outside of ORACLE_HOME, in our case /u02
[oracle@linux75-2 122]$ pwd
/u01/orcl/122
[oracle@linux75-2 122]$ mkdir -p /u02/.patch_storage
[oracle@linux75-2 122]$ ln -s /u02/.patch_storage .patch_storage
[oracle@linux75-2 122]$ ls -lart .pat*
lrwxrwxrwx. 1 oracle oinstall 19 Sep  5 23:41 .patch_storage -> /u02/.patch_storage
[oracle@linux75-2 122]$
Now, the patching completes without any issues as below. 

[oracle@linux75-2 32916808]$ $ORACLE_HOME/OPatch/opatch apply OPatch.SKIP_VERIFY_SPACE=true
Oracle Interim Patch Installer version 12.2.0.1.26
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/orcl/122
Central Inventory : /u01/oraInventory
   from           : /u01/orcl/122/oraInst.loc
OPatch version    : 12.2.0.1.26
OUI version       : 12.2.0.1.4
Log file location : /u01/orcl/122/cfgtoollogs/opatch/opatch2021-09-05_23-43-32PM_1.log

Verifying environment and performing prerequisite checks...

--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...

Oracle Interim Patch Installer version 12.2.0.1.26
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/orcl/122
Central Inventory : /u01/oraInventory
   from           : /u01/orcl/122/oraInst.loc
OPatch version    : 12.2.0.1.26
OUI version       : 12.2.0.1.4
Log file location : /u01/orcl/122/cfgtoollogs/opatch/opatch2021-09-05_23-43-46PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   32916808

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/orcl/122')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '32916808' to OH '/u01/orcl/122'
ApplySession: Optional component(s) [ oracle.swd, 12.2.0.1.0 ] , [ oracle.swd.oui, 12.2.0.1.0 ] , [ oracle.network.cman, 12.2.0.1.0 ] , [ oracle.ons.cclient, 12.2.0.1.0 ] , [ oracle.ons.eons.bwcompat, 12.2.0.1.0 ] , [ oracle.rdbms.drdaas, 12.2.0.1.0 ] , [ oracle.oid.client, 12.2.0.1.0 ] , [ oracle.ons.daemon, 12.2.0.1.0 ] , [ oracle.network.gsm, 12.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 12.2.0.1.0...

Patching component oracle.rdbms.rsf, 12.2.0.1.0...

Patching component oracle.network.rsf, 12.2.0.1.0...

Patching component oracle.rdbms.util, 12.2.0.1.0...

Patching component oracle.ctx, 12.2.0.1.0...

...

...

...

Patching component oracle.jdk, 1.8.0.91.0...

OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of ‘/u01/orcl/122/bin/extjobO’: Operation not permitted
make: [iextjob] Error 1 (ignored)


Patch 32916808 successfully applied.
OPatch Session completed with warnings.
Log file location: /u01/orcl/122/cfgtoollogs/opatch/opatch2021-09-05_23-43-46PM_1.log

OPatch completed with warnings.
[oracle@linux75-2 32916808]$ 
Note that the opatch is started using OPatch.SKIP_VERIFY_SPACE=true
The final error can be ignored or can be fixed by running root.sh from ORACLE_HOME as root user. 
We are now done with the patching with limited storage in Oracle home file system. 

Foot note: 

The file system utilization of /u02 increased up to 14 GB (from 9.7 GB) as expected and went back to 11 GB and not completely down to 9.7 GB (using ~ 1GB post patching). 
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G  9.7G  7.3G  58% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   12G  5.2G  70% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   13G  4.2G  76% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   13G  4.1G  76% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   14G  3.9G  78% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   14G  3.8G  78% /u02
[oracle@linux75-2 u02]$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        17G   11G  6.5G  62% /u02
 
This is because OPatch utility will delete all the backup files which are not required post patching and keeps any file that will be needed for rollback of the patch if intended. Check MOS Doc ID 550522.1 to understand how to clean up .patch_storage if needed. 

References: 







Happy Patching...!!!