LVM on Linux demo to add disk and create mount points

 Logical Volume Management (LVM) is very powerful device mapper framework that provides logical volume management for Linux systems. 

In this post I'll just present you the minimal commands that are required to make use of the storage that is added to OS. In my last post, I've written about how to add storage to guest OS running on a VirtualBox. It can be compared to the storage addition done at OS level in companies. Once the storage is provided to the OS (be it a single disk added or a chunk from SAN, etc.,), we will look at how to make the storage available at the Linux OS for usage. 

The process will involve the below steps

• Create Linux LVM partition using fdisk
• Create physical volume
• Create Volume group
• Create Logical Volume
• Create file system (ext4 used in this example)
• Mount the FS
• Make the FS mount permanent

Explanation of commands (highlighted lines) used are provided as comment lines prior to the commands in the demo. A few notable result lines are also highlighted. 

 
[root@12r1-rac2 ~]# #check for available File systems
[root@12r1-rac2 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  6.0G   18G  26% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  267G  234G  54% /media/sf_Oracle
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #List disks available, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

...
...
...
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056a91

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3264    26214400   83  Linux
/dev/sda2            3264        4830    12579840   83  Linux
/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Format the newly added disk. 
[root@12r1-rac2 ~]# #I'm creating 2 primary partitions with 15GB each in total of 30GB disk
[root@12r1-rac2 ~]# #Using n,p,1,1,+15G,n,p,2(default),1960(default),3916(default),w as input
[root@12r1-rac2 ~]# #This step can be skipped if you want to partition while initializing using pvcreate
[root@12r1-rac2 /]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3916, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-3916, default 3916): +15G

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1960-3916, default 1960):
Using default value 1960
Last cylinder, +cylinders or +size{K,M,G} (1960-3916, default 3916):
Using default value 3916

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 /]#
[root@12r1-rac2 ~]# #List the available disks. Newly created disks are visible, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   83  Linux
/dev/sdf2            1960        3916    15719602+  83  Linux
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Change the partition type of /dev/sdf1, /dev/sdf2 to Linux LVM 
[root@12r1-rac2 ~]# #Using t,1,8e,t,2,8e,w as input
[root@12r1-rac2 ~]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the available disks, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   8e  Linux LVM
/dev/sdf2            1960        3916    15719602+  8e  Linux LVM
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Currently LVM is not configured in the system and hence no o/p for the below 3 commands
[root@12r1-rac2 ~]# pvs
[root@12r1-rac2 ~]# vgs
[root@12r1-rac2 ~]# lvs
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#	#Create physical volumes using available disks. (also used to format block device if not formatted already)
[root@12r1-rac2 ~]# pvcreate /dev/sdf1 /dev/sdf2
  Physical volume "/dev/sdf1" successfully created
  Physical volume "/dev/sdf2" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the physical volumes, you can use 'pvdisplay' command to view detailed information
[root@12r1-rac2 ~]# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sdf1       lvm2 ---  15.01g 15.01g
  /dev/sdf2       lvm2 ---  14.99g 14.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Create Volume Group (named vg18c) using the available PVs 
[root@12r1-rac2 ~]# vgcreate vg18c /dev/sdf1 /dev/sdf2
  Volume group "vg18c" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List available VGs, #PV denotes 2 PVs associated with this VG. 
[root@12r1-rac2 ~]# #Use 'vgdisplay' command to view detailed information
[root@12r1-rac2 ~]# vgs
  VG    #PV #LV #SN Attr   VSize  VFree
  vg18c   2   0   0 wz--n- 29.99g 29.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Create Logical Volume from the available VG
[root@12r1-rac2 ~]# # -L is size and -n is name. Last variable is the VG, the LV will be created from.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cgrid vg18c
  Logical volume "18cgrid" created.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cdb vg18c
  Logical volume "18cdb" created.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the LVs. Use 'lvdisplay' to view detailed information
[root@12r1-rac2 ~]# lvs
  LV      VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  18cdb   vg18c -wi-a----- 12.00g
  18cgrid vg18c -wi-a----- 12.00g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Now create File system of type ext4 with the available LV using mkfs command
[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cdb
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cgrid
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# mkdir /oradir1 /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Mount the FS using mount command
[root@12r1-rac2 ~]# mount /dev/vg18c/18cgrid /oradir1
[root@12r1-rac2 ~]# mount /dev/vg18c/18cdb /oradir2
[root@12r1-rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  6.0G   18G  26% /
tmpfs                 2.0G  629M  1.4G  32% /dev/shm
/dev/sda3             9.8G  5.9G  3.4G  64% /u01
/dev/sda2              12G  4.9G  6.3G  44% /u02
Oracle                500G  267G  234G  54% /media/sf_Oracle
/dev/mapper/vg18c-18cgrid
                       12G   30M   12G   1% /oradir1
/dev/mapper/vg18c-18cdb
                       12G   30M   12G   1% /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Edit the fstab to make the mount permanent across reboots. 
[root@12r1-rac2 ~]# vi /etc/fstab
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Oct 20 01:50:26 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b149969c-329d-48d7-ace0-ea7ac6671964 /                       ext4    defaults        1 1
UUID=aa9b0bed-4eab-407d-8c0f-c2739a7678fb /u01                    ext4    defaults        1 2
UUID=2ab60579-5059-4c1e-9db1-7ac8d9a5ac77 /u02                    ext4    defaults        1 2
UUID=6ebf30ec-79d9-4c54-8e44-ce7f47190fdd swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/mapper/vg18c-18cgrid     /oradir1                ext4    defaults        0 0
/dev/mapper/vg18c-18cdb       /oradir2                ext4    defaults        0 0
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# 

If you would like to remove the LV, VG and PV, you can do the following

[root@12r1-rac2 /]# #Make sure to umount the file system. 
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cgrid
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cdb
[root@12r1-rac2 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  5.1G   19G  22% /
tmpfs           2.0G  633M  1.4G  32% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  268G  233G  54% /media/sf_Oracle
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the logical volume
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cgrid
Do you really want to remove active logical volume 18cgrid? [y/n]: y
  Logical volume "18cgrid" successfully removed
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cdb
Do you really want to remove active logical volume 18cdb? [y/n]: y
  Logical volume "18cdb" successfully removed
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#
[root@12r1-rac2 /]# #Remove the volume group
[root@12r1-rac2 /]# vgremove vg18c
  Volume group "vg18c" successfully removed
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the Physical volumes created
[root@12r1-rac2 /]# pvremove /dev/sdf1 /dev/sdf2
  Labels on physical volume "/dev/sdf1" successfully wiped
  Labels on physical volume "/dev/sdf2" successfully wiped
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#

References: 

Logical Volume Manager Administration Red Hat Enterprise Linux 6 | Red Hat Customer Portal

How to create a physical volume in Linux using LVM | Enable Sysadmin (redhat.com)

Common Tasks (tldp.org)

Happy LVMing...!!!

Adding additional disk to guest OS on VirtualBox

 If you are running out of space on the guest OS that you have created under the VirtualBox, the following steps can be followed to add a new disk. I have provided the steps in GUI which can also be done using VBoxManage createmedium command. 

Step 1: Go to File - Click on Virtual Media Manager

Step 2: Click on Create

Step 3: Choose VMDK and click on Next

To learn about different Hard disk file type, click here

Step 4: Choose Dynamically allocated and click Next

Step 5: Provide the image file name, size and click on Create

Since we have provided option to Dynamically allocated, the disk image will be created and the initial size will be very low. I have provided 30 GB of size and the initial disk image created is of size 3.81 MB which will grow dynamically as we start using/writing on it. 

Click on close once the disk is created successfully. 

Once the disk image is created, we can then attach the storage to the OS that we are in need of space as below. The same operation can also be done using command line with VBoxManage storageattach commands

Step 1: Select the VM you want to add the disk and click on Storage

Step 2: Click on Adds Hard disk button under Controller: SATA

Step 3: Choose the hard disk you would like to add and click on Choose

You can now see that the hard disk is added to the VM. Click on OK to close the dialog box and start the VM to use. 

Happy learning...!!!

Configuring TCPS connection, Jan 2021 DBRU (Oracle 19.10) and resolving ORA-28864, TNS-01194

In one of our DB environment, we use TCPS only connection defined in the listener configuration

which used to run properly without any issues. The listener.ora and sqlnet.ora files are as below.

[oracle@abc]$ more listener.ora
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.195.60.243)(PORT = 2484))
      #(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 10.195.60.243)(PORT = 1521))
    )
  )

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = ABC)
      (ORACLE_HOME = /oracle/product/base/19.3.0.0)
      (SID_NAME = ABC)
    )
  )

LOGGING_LISTENER = OFF
SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF
INBOUND_CONNECT_TIMEOUT_LISTENER = 0
SECURE_CONTROL_LISTENER = TCPS
SECURE_REGISTER_LISTENER = TCPS
WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /oracle/product/base/server_wallet)
    )
  )
admin_restrictions_LISTENER = ON
SSL_CLIENT_AUTHENTICATION = FALSE
ADR_BASE=/home/oracle/trace
TRACE_LEVEL_LISTENER=10
TRACE_DIRECTORY_LISTENER=/home/oracle/trace
TRACE_FILE_LISTENER=listener.log
[oracle@abc]$ 

[oracle@abc]$ more sqlnet.ora
SQLNET.INBOUND_CONNECT_TIMEOUT=0
SQLNET.AUTHENTICATION_SERVICES=(BEQ,TCPS)
WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /oracle/product/base/server_wallet)
    )
  )
SSL_CLIENT_AUTHENTICATION = FALSE
[oracle@abc]$ 

You can take a look here to understand what the 2 highlighted parameters would do. Basically, we are trying to secure the listener connections and also restrict client connections to TCPS protocol and this configuration was working fine until we patch the database from Oracle version 19.9 to 19.10. 

Upon starting the listener post 19.10 (Jan 21 RU) patching, we are getting the below error

[oracle@abc]$ lsnrctl start

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 09-APR-2021 11:43:36

Copyright (c) 1991, 2020, Oracle.  All rights reserved.

Starting /oracle/product/base/19.3.0.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /oracle/product/base/19.3.0.0/network/admin/listener.ora
Trace information written to /oracle/product/base/diag/tnslsnr/server1/listener/trace/ora_3647_140266612784320.trc
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.195.60.243)(PORT=2484)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=10.195.60.243)(PORT=2484)))
ORA-28864: SSL connection closed gracefully
 TNS-12560: TNS:protocol adapter error
  TNS-00542: SSL Handshake failed
   Linux Error: 28864: Unknown error 28864
[oracle@abc]$ 

Oracle support note Doc ID 2280787.1 provides the steps to properly configure and restrict the Lsnrctl Utility to Communicate with the Listener Using ONLY the TCPS Protocol which was exactly followed in our environment. 

We even recreated the wallet as below so as to be sure that there are no mistakes in the process followed. 

[oracle@abc]$ orapki wallet create -wallet /oracle/product/base/server_wallet -auto_login -pwd ********
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
[oracle@abc]$ orapki wallet add -wallet /oracle/product/base/server_wallet -dn "CN=server" -keysize 512 -self_signed -validity 3650 -pwd ********
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
[oracle@abc]$ orapki wallet add -wallet /oracle/product/base/server_wallet -trusted_cert -cert /temp/devid_abc_com.crt
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Cannot modify auto-login (sso) wallet
Enter wallet password:
Operation is successfully completed.
[oracle@abc]$ 
[oracle@abc]$ cd server_wallet
[oracle@abc]$ ls -lrt
total 12
-rw-------. 1 oracle oinstall    0 Apr  9 11:12 ewallet.p12.lck
-rw-------. 1 oracle oinstall    0 Apr  9 11:12 cwallet.sso.lck
-rwxr-x---. 1 oracle oinstall 2326 Apr  9 11:12 devid_abc_com.crt
-rw-------. 1 oracle oinstall 3440 Apr  9 11:13 ewallet.p12
-rw-------. 1 oracle oinstall 3485 Apr  9 11:13 cwallet.sso
[oracle@abc]$ orapki wallet display -wallet /oracle/product/base/server_wallet
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=server
Trusted Certificates:
Subject:        CN=server
Subject:        CN=devid.abc.com,O=xxxx International Inc.,L=Morris Plains,ST=New Jersey,C=US
[oracle@abc]$ 

Even after recreating the wallet and listener.ora and sqlnet.ora files are properly configured, we still hit the same error while starting the listener. 

Things got interesting when the patches were rolled back to 19.9, the listener started without issues and application was able to connect via TCPS connectivity. 

So we raised an SR suspecting the issue to be a bug with Jan 21 RU and the ticket is work in progress. Oracle support works on their own pace and our investigation continues. 

I came across Oracle support note Doc ID 2732618.1 when I was trying to keep only the TCPS connection in the listener and start to drill down the issue. The note suggested to check the validity of the certificate in the wallet which might have expired. You might have seen in our wallet configuration, we also added 3rd party certificate from client for their secure connection. We can't suspect the client certificate here as the listener itself is not starting in the first place and the server wallet certificate is also self signed just now with validity of 3650 days. So what could actually be wrong? 

Oracle support note Doc ID 2674842.1 explains the procedure on how to check the validity of the certificate which is as show below. 

[oracle@abc]$ orapki wallet export -wallet /oracle/product/base/server_wallet -dn 'CN=server' -cert wallet.cert
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
[oracle@abc]$ 
[oracle@abc]$ $ORACLE_HOME/jdk/jre/bin/keytool -printcert -file wallet.cert
Owner: CN=server
Issuer: CN=server
Serial number: aa005d7cd120d6588e8540f95914c3f
Valid from: Fri Apr 09 11:12:22 UTC 2021 until: Mon Apr 07 11:12:22 UTC 2031
Certificate fingerprints:
         SHA1: 25:AB:2D:6D:39:A6:84:2E:8D:14:9A:83:66:F5:E6:5E:11:FC:B2:8F
         SHA256: 83:19:46:95:7D:EE:35:08:0F:86:A8:77:AE:F7:F8:E1:27:F4:75:33:5D:F0:26:9A:3C:D4:36:BE:51:49:E4:D7
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 512-bit RSA key (disabled)
Version: 1

Warning:
The certificate uses a 512-bit RSA key which is considered a security risk and is disabled.
[oracle@abc]$ 

Now, we are able to find the root cause of the issue. The certificate key is disabled and also a warning is printed stating the same. 

The solution now is easy, change the keysize to 1024 or 2048 while signing the certificate if you don't want to get into this trouble again. 

[oracle@abc]$ orapki wallet remove -wallet /oracle/product/base/server_wallet -dn 'CN=server'
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Cannot modify auto-login (sso) wallet
Enter wallet password:
Operation is successfully completed.
[oracle@abc]$ 
[oracle@abc]$ orapki wallet add -wallet /oracle/product/base/server_wallet -dn "CN=server" -keysize 1024 -self_signed -validity 3650 -pwd ********
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2020, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
[oracle@abc]$ 
[oracle@abc]$ lsnrctl start

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 09-APR-2021 12:06:51

Copyright (c) 1991, 2020, Oracle.  All rights reserved.

Starting /oracle/product/base/19.3.0.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /oracle/product/base/19.3.0.0/network/admin/listener.ora
Trace information written to /oracle/product/base/diag/tnslsnr/server1/listener/trace/ora_10336_140143471860928.trc
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.195.60.243)(PORT=2484)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.195.60.243)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=10.195.60.243)(PORT=2484)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                09-APR-2021 12:06:51
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               admin
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/product/base/19.3.0.0/network/admin/listener.ora
Listener Trace File       /oracle/product/base/diag/tnslsnr/server1/listener/trace/ora_10336_140143471860928.trc
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.195.60.243)(PORT=2484)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.195.60.243)(PORT=1521)))
Services Summary...
Service "ABC" has 1 instance(s).
  Instance "ABC", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@abc]$ 

I have checked in support notes and also in the known issues, bugs fixed or read me file of the Jan 21 DBRU in which I couldn't find any mention of this restriction that keysize with 512 bits are not supported. All the documents that I have referred in the Oracle support site mentions the keysize as 512 bit which needs to be updated if it is not supported w.r.to a specific version and above patching. 

So the take away from this issue and resolution is that when you are planning for Oracle database 19.10 patching, check whether the certificates are signed with 1024 bits or higher. 

Update:

Oracle support resolved the ticket after we shared with them that we used a keysize 1024 instead of 512. Still don't know where is this limitation documented :)

References: 

Oracle Net Listener Parameters in the listener.ora File

How to Restrict the Lsnrctl Utility to Communicate with the Listener Using ONLY the TCPS Protocol [2280787.1]

Listener Fails to Start Using Only TCPS Address - ORA-28864: SSL Connection Closed Gracefully [2732618.1]

How to Check Validity of Server Certificate from ORAPKI Wallet in Command Line [2674842.1]

Understanding and Investigating SSL Issues Support Pattern [1078957.1]

Database 19 Release Updates and Revisions Bugs Fixed Lists [2523220.1]

Happy Troubleshoooting...!!!