LVM on Linux demo to add disk and create mount points

 Logical Volume Management (LVM) is very powerful device mapper framework that provides logical volume management for Linux systems. 

In this post I'll just present you the minimal commands that are required to make use of the storage that is added to OS. In my last post, I've written about how to add storage to guest OS running on a VirtualBox. It can be compared to the storage addition done at OS level in companies. Once the storage is provided to the OS (be it a single disk added or a chunk from SAN, etc.,), we will look at how to make the storage available at the Linux OS for usage. 

The process will involve the below steps

• Create Linux LVM partition using fdisk
• Create physical volume
• Create Volume group
• Create Logical Volume
• Create file system (ext4 used in this example)
• Mount the FS
• Make the FS mount permanent

Explanation of commands (highlighted lines) used are provided as comment lines prior to the commands in the demo. A few notable result lines are also highlighted. 

 
[root@12r1-rac2 ~]# #check for available File systems
[root@12r1-rac2 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  6.0G   18G  26% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  267G  234G  54% /media/sf_Oracle
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #List disks available, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

...
...
...
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056a91

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3264    26214400   83  Linux
/dev/sda2            3264        4830    12579840   83  Linux
/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Format the newly added disk. 
[root@12r1-rac2 ~]# #I'm creating 2 primary partitions with 15GB each in total of 30GB disk
[root@12r1-rac2 ~]# #Using n,p,1,1,+15G,n,p,2(default),1960(default),3916(default),w as input
[root@12r1-rac2 ~]# #This step can be skipped if you want to partition while initializing using pvcreate
[root@12r1-rac2 /]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3916, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-3916, default 3916): +15G

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1960-3916, default 1960):
Using default value 1960
Last cylinder, +cylinders or +size{K,M,G} (1960-3916, default 3916):
Using default value 3916

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 /]#
[root@12r1-rac2 ~]# #List the available disks. Newly created disks are visible, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   83  Linux
/dev/sdf2            1960        3916    15719602+  83  Linux
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Change the partition type of /dev/sdf1, /dev/sdf2 to Linux LVM 
[root@12r1-rac2 ~]# #Using t,1,8e,t,2,8e,w as input
[root@12r1-rac2 ~]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the available disks, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   8e  Linux LVM
/dev/sdf2            1960        3916    15719602+  8e  Linux LVM
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Currently LVM is not configured in the system and hence no o/p for the below 3 commands
[root@12r1-rac2 ~]# pvs
[root@12r1-rac2 ~]# vgs
[root@12r1-rac2 ~]# lvs
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#	#Create physical volumes using available disks. (also used to format block device if not formatted already)
[root@12r1-rac2 ~]# pvcreate /dev/sdf1 /dev/sdf2
  Physical volume "/dev/sdf1" successfully created
  Physical volume "/dev/sdf2" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the physical volumes, you can use 'pvdisplay' command to view detailed information
[root@12r1-rac2 ~]# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sdf1       lvm2 ---  15.01g 15.01g
  /dev/sdf2       lvm2 ---  14.99g 14.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Create Volume Group (named vg18c) using the available PVs 
[root@12r1-rac2 ~]# vgcreate vg18c /dev/sdf1 /dev/sdf2
  Volume group "vg18c" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List available VGs, #PV denotes 2 PVs associated with this VG. 
[root@12r1-rac2 ~]# #Use 'vgdisplay' command to view detailed information
[root@12r1-rac2 ~]# vgs
  VG    #PV #LV #SN Attr   VSize  VFree
  vg18c   2   0   0 wz--n- 29.99g 29.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Create Logical Volume from the available VG
[root@12r1-rac2 ~]# # -L is size and -n is name. Last variable is the VG, the LV will be created from.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cgrid vg18c
  Logical volume "18cgrid" created.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cdb vg18c
  Logical volume "18cdb" created.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the LVs. Use 'lvdisplay' to view detailed information
[root@12r1-rac2 ~]# lvs
  LV      VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  18cdb   vg18c -wi-a----- 12.00g
  18cgrid vg18c -wi-a----- 12.00g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Now create File system of type ext4 with the available LV using mkfs command
[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cdb
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cgrid
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# mkdir /oradir1 /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Mount the FS using mount command
[root@12r1-rac2 ~]# mount /dev/vg18c/18cgrid /oradir1
[root@12r1-rac2 ~]# mount /dev/vg18c/18cdb /oradir2
[root@12r1-rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  6.0G   18G  26% /
tmpfs                 2.0G  629M  1.4G  32% /dev/shm
/dev/sda3             9.8G  5.9G  3.4G  64% /u01
/dev/sda2              12G  4.9G  6.3G  44% /u02
Oracle                500G  267G  234G  54% /media/sf_Oracle
/dev/mapper/vg18c-18cgrid
                       12G   30M   12G   1% /oradir1
/dev/mapper/vg18c-18cdb
                       12G   30M   12G   1% /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Edit the fstab to make the mount permanent across reboots. 
[root@12r1-rac2 ~]# vi /etc/fstab
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Oct 20 01:50:26 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b149969c-329d-48d7-ace0-ea7ac6671964 /                       ext4    defaults        1 1
UUID=aa9b0bed-4eab-407d-8c0f-c2739a7678fb /u01                    ext4    defaults        1 2
UUID=2ab60579-5059-4c1e-9db1-7ac8d9a5ac77 /u02                    ext4    defaults        1 2
UUID=6ebf30ec-79d9-4c54-8e44-ce7f47190fdd swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/mapper/vg18c-18cgrid     /oradir1                ext4    defaults        0 0
/dev/mapper/vg18c-18cdb       /oradir2                ext4    defaults        0 0
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# 

If you would like to remove the LV, VG and PV, you can do the following

[root@12r1-rac2 /]# #Make sure to umount the file system. 
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cgrid
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cdb
[root@12r1-rac2 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  5.1G   19G  22% /
tmpfs           2.0G  633M  1.4G  32% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  268G  233G  54% /media/sf_Oracle
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the logical volume
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cgrid
Do you really want to remove active logical volume 18cgrid? [y/n]: y
  Logical volume "18cgrid" successfully removed
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cdb
Do you really want to remove active logical volume 18cdb? [y/n]: y
  Logical volume "18cdb" successfully removed
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#
[root@12r1-rac2 /]# #Remove the volume group
[root@12r1-rac2 /]# vgremove vg18c
  Volume group "vg18c" successfully removed
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the Physical volumes created
[root@12r1-rac2 /]# pvremove /dev/sdf1 /dev/sdf2
  Labels on physical volume "/dev/sdf1" successfully wiped
  Labels on physical volume "/dev/sdf2" successfully wiped
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#

References: 

Logical Volume Manager Administration Red Hat Enterprise Linux 6 | Red Hat Customer Portal

How to create a physical volume in Linux using LVM | Enable Sysadmin (redhat.com)

Common Tasks (tldp.org)

Happy LVMing...!!!

Ansible ping and types of Unix shells, a learning...

 I've just started to learn the basics of Ansible which is a powerful automation tool. The first step of any automation is to setup the environment on which we are working. I'm referring Frits Hoogland's blog which is very easy to understand the basics of Ansible. 

The first step is to set up your hosts file and test the connectivity between the servers. We have an inbuilt Ansible ping module to do this. Here is how my hosts file looks like 

[oracle@linux-8 ansible-project]$ cat hosts
192.168.56.131
192.168.56.151

[l7]
192.168.56.151

[l8]
192.168.56.131
192.168.56.181

[oracle@linux-8 ansible-project]$

I'm using the below command to test the ping and the result is also provided. Just keep in mind I have already setup password less communication and hence not required to pass the password. If I still need to prompt for password, I can provide -k option to the command line. 

[oracle@linux-8 ansible-project]$ ansible l8 -m ping -u oracle
SSH password:
[WARNING]: Unhandled error in Python interpreter discovery for host 192.168.56.131: unexpected output from Python interpreter discovery
192.168.56.131 | FAILED! => {
    "msg": "failed to transfer file to /home/oracle/.ansible/tmp/ansible-local-32681q1m7n_gx/tmpzvfmvwgz /home/oracle/.ansible/tmp/ansible-tmp-1617045011.6527164-32687-33868194395919/AnsiballZ_ping.py:\n\nOracle Home set as /u01/db/122 and SID=orcl\n"
}
192.168.56.181 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[oracle@linux-8 ansible-project]$

We can see the ping failed for 192.168.56.131 server with the error line including "\n\nOracle Home set as /u01/db/122 and SID=orcl\n"

This "\n\nOracle Home set as /u01/db/122 and SID=orcl\n" is a environment info message I have set in the server when we login as Oracle user using .bashrc file. 

Normal ssh works without any issues.

[oracle@linux-8 ansible-project]$ ssh 192.168.56.131
Web console: https://linux75.selvapc.com:9090/ or https://10.0.2.15:9090/

Last login: Mon Mar 29 14:46:24 2021 from 192.168.56.181
Oracle Home set as /u01/db/122 and SID=orcl
[oracle@linux75 ~]$

We can see the info message line once we login. 

All that Ansible ping command does is try to connect to host, verify a usable python and return pong on success. So, the question here is why my ssh command works without any issues but not my Ansible ping. 

The answer lies in how the connection is being established to the server. There are 4 types of shells in Unix. 

• Interactive login                     - Login session started using graphical interface or by providing username/password via ssh
• Interactive non-login             - Sessions started by opening another terminal in a server
• Non interactive non-login     - Sessions started by scripts execution where the commands are run and exits the shell. We can't interact with these shells
• Non interactive login     - Providing a command to run using ssh like "echo "some command" |ssh servername". Here instead of login, the session will run the "some command" and exits

So now, the difference in these shells is that the Login shells (shells started by the login system process, or by your X display manager) uses or reads .profile or .login files to set your environment or initialize your environment

Non interactive login/non-login shells, such as scripts started by shell or the Ansible ping that we are using now reads .bashrc or .kshrc files to initialize environment. Hence we should not set up any echo statements which would then become stdout to the shell initialization (which the shell won't expect) and errors out as a result. 

My .bashrc file looked like the below which was causing the issue with the non interactive session from Ansible ping. 

[oracle@linux75 ~]$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
export ORACLE_BASE=/u01/db
export ORACLE_HOME=/u01/db/122
export PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_SID=orcl
echo "Oracle Home set as $ORACLE_HOME and SID=$ORACLE_SID"

Once I remove the "echo" line from the file, I didn't have issues from the Ansible ping. We can simply move the echo line to the .bash_profile (I already had the entry in the .bash_profile file as well which is why when I previously did a ssh we got the info message once we login) which is the file used to initialize when we start interactive session. 

[oracle@linux-8 ansible-project]$ ansible l8 -m ping -u oracle -k
SSH password:
192.168.56.181 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
192.168.56.131 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[oracle@linux-8 ansible-project]$

You can notice now I have used -k option and the password is being prompted. Ping works as expected on all the servers in the group l8 and now we can proceed further with our automations..

Hope you have learnt regarding different types of Unix shells and the different initialization files they use. 

References: 

what do you mean by interactive shell? - Unix & Linux Stack Exchange

bash - What is the purpose of .bashrc and how does it work? - Unix & Linux Stack Exchange

bash - Differentiate Interactive login and non-interactive non-login shell - Ask Ubuntu

Happy Shelling!!! :)

Java HeadlessException on generating OSWatcher Analyzer graphs

 OS Watcher is a very handy tool in inspecting/diagnosing and troubleshooting any performance issues on the unix platforms. It is advised to run OSWatcher on every node that the Oracle instance is running and in the case of a performance issue, Oracle support can use this data to help diagnose performance problems which may outside the database. 

Installation and usage notes can be found with the following support notes

• OSWatcher (Includes: [Video]) (Doc ID 301137.1)
• OS Watcher User's Guide (Doc ID 1531223.1)
• OSWatcher Analyzer User Guide (Doc ID 461053.1)

To outline, installation is very simple. One can download the latest version of OS Watcher Black Box or OSWbb (V8.4.0 at the time of writing this post) and transfer it to the server where the tool needs to be installed and untar the contents to the desired directory.

$ tar xvf oswbb.tar
$ chmod 744 *

The tool can be started by the below command

$ ./startOSWbb.sh 15 48 NONE /media/sf_Oracle/software/oswlinux-8

• ARG1 = snapshot interval in seconds.
• ARG2 = the number of hours of archive data to store.
• ARG3 = (optional) the name of a compress utility to compress each file automatically after it is created.
• ARG4 = (optional) an alternate (non default) location to store the archive directory.

OS Watcher Analyzer (oswbba) is java based utility that allows the user to graph and analyze data collected from running OSWatcher and it comes bundled with the oswbb tar file and no separate installation is needed. 

The latest version required java version 8 and higher. So make sure you have java with version 8 or higher is installed before running oswbba. 

oswbba can be started with the below command which prompts you for multiple options to display various graphs or to generate gifs or complete dashboard. 

$ java -jar oswbba.jar -i /location/of/archives 

This post is to cover the error encountered while starting the oswbba utility w.r.to java and how to overcome the same. 

[oracle@linux-8 oswbb]$ pwd
/oracle/osw/oswbb
[oracle@linux-8 oswbb]$ which java
/usr/bin/java
[oracle@linux-8 oswbb]$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
[oracle@linux-8 oswbb]$

I have already installed oswbb tool and I do have required java version installed as well.

While starting the tool and trying to generate the report, I encountered the below error. 

[oracle@linux-8 oswbb]$ java -jar oswbba.jar -i /media/sf_Oracle/software/oswlinux-8

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c)  2019 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...


Parsing file linux-8.selvapc.com_pidstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_pidstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_iostat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_iostat_21.03.02.0100.dat ...
This directory already exists. Rewriting...

Parsing file linux-8.selvapc.com_vmstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_vmstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_netstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_netstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_top_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_top_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_ps_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_ps_21.03.02.0100.dat ...


Parsing Completed.


Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter 61 to Display Individual OS Process I/O RPS Graphs
Enter 62 to Display Individual OS Process I/O WPS Graphs
Enter 63 to Display Individual OS Process Percent User CPU Graphs
Enter 64 to Display Individual OS Process Percent System CPU Graphs
Enter 65 to Display Individual OS Process Percent Total CPU (User + System) Graphs
Enter 66 to Display Individual OS Process Percent Memory Graphs

Enter GP to Generate Individual Process Profile
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files

Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
Enter B to Returns to Baseline Graph Time Scale (Does not change analysis dataset)
Enter R to Remove Currently Displayed Graphs

Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:1
>>> Input Error: null
java.awt.HeadlessException
        at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
        at java.awt.Window.<init>(Window.java:536)
        at java.awt.Frame.<init>(Frame.java:420)
        at java.awt.Frame.<init>(Frame.java:385)
        at k.c(Unknown Source)
        at OSWGraph.OSWGraph.main(Unknown Source)
[oracle@linux-8 oswbb]$

There are 2 options to overcome this error. 

Option 1: 

If the tool is being run on a server with Oracle database software installed, we can use the java software that comes along with the database software. Make sure the java version is 8 or higher. 

* Output truncated for easy readability

[oracle@linux-8 oswbb]$ $ORACLE_HOME/jdk/bin/java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
[oracle@linux-8 oswbb]$
[oracle@linux-8 oswbb]$ $ORACLE_HOME/jdk/bin/java -jar oswbba.jar -i /media/sf_Oracle/software/oswlinux-8

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c)  2019 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...


Parsing file linux-8.selvapc.com_pidstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_pidstat_21.03.02.0100.dat ...

...
...
...
Parsing file linux-8.selvapc.com_ps_21.03.02.0100.dat ...


Parsing Completed.


Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
...
...
...
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:1

Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
...
...
...
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:

Now, we don't see that error and the tool generated the graph and the output is displayed on the screen.

Sample graph below

Note: We need to have X11 forwarding enabled to get the output, else we will encounter the below error

Please Select an Option:1
>>> Input Error:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.

Option 2: 

If we are installing the oswbb tool on a server where there is no Oracle database software installed or if the bundled java along with the database software is of lower version, check the java package that is already installed on the server. 

By default, on RHEL 7 or RHEL 8, the OpenJDK software comes installed only with the headless package. 

[oracle@linux-8 oswbb]$ rpm -qa|grep java-1.8

java-1.8.0-openjdk-headless-1.8.0.282.b08-2.el8_3.x86_64

We can install java-1.8.0-openjdk package additional to the headless package and start running the oswbba tool to analyze the data collected to investigate performance issues. 

As I have run the tool on a server where Oracle software is installed, I used option 1 to overcome the error. 

Reference: 

• swing - All my java applications now throw a java.awt.headlessexception - Stack Overflow
• Using Headless Mode in the Java SE Platform (oracle.com)
• Picture courtesy: www.oracle.com

Happy troubleshooting...!!!

RHEL 7 to RHEL 8 upgrade on an Oracle RDBMS installed server

 In my last post, I've converted my Oracle Linux to Red Hat Enterprise Linux and we saw that the database was running without any issues post conversion as well. The main reason I did the conversion is to learn a few things from Linux server administration and also I need a RHEL 7 server hosting Oracle database to test the upgrade from RHEL 7 to RHEL 8. This upgrade is officially supported by Red Hat. 

In this post I'll be following exactly what is provided in the below official Red Hat link which is a straight forward step by step procedure where I'll discuss what all things needs to be taken care and how the upgrade goes and importantly to test the functionality of Oracle database running on the upgraded server post upgrade. 

The step by step procedure is outlined in the below link 

Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Unlike the previous RHEL 6 to RHEL 7 upgrade document (though not officially supported), this document usage is very easy to follow and there seems to be no hiccups anywhere even for a beginner in System Administration. So let's get to our observation and results. 

1. You need to be on latest available version, currently 7.9 

2. Third party apps can also be migrated to upgraded version using custom actors. (I've not done this as I don't have any)

3. During upgrade process, Leapp utility sets SELinux mode to permissive post which user needs to change it to enforcing. 

4. There is a slightly different method of upgrade if you run SAP HANA (we are not discussing here)

5. There are a few limitations as well. So understand these before planning for the upgrade. 

6. If you don't have a satellite server or Red Hat subscription, you can also setup custom repo containing at least latest rpms of rhel 7 server, rhel 7 server extras, rhel 8 base OS and rhel 8 appstream

7. Install leapp utility along with leapp-repository and cockpit-leapp 

The main aspect of this upgrade is the pre upgrade report provided by the leapp utility. 

Run the "leapp preupgrade" command either from a terminal or from the server web console to identify potential problems associated with the upgrade and their remediation methods for a smoother upgrade process. 

For Linux admin beginners (like me), web console is a very attractive, easy to use console accessible via web using the https://localhost:9090 or https:(servername)or(ip):9090 with root or user with sudo permissions. 

Sample window looks like the below

In my system, I've ran the preupgrade and below are the observations. Once after running the leapp preupgrade command, we can click on the "Upgrade Report" on the left hand side of the console to view the report with its severity or Risk factor, information messages, remediation plans and inhibitors.

Sample report of my test system

We can see there are 6 Titles reported as High Risk factor in which 3 are Inhibitors. Inhibitors needs to be remediated and if not, upgrade will not progress. Let's take a look at each of them. 

1. Disable e1000 driver

First inhibitor in the report is the unsupported driver found in the system e1000. e1000 is the network driver installed when choosing Intel PRO/1000 MT desktop as network adaptor while configuring the VM on VirtualBox. Simple fix is to use a different adaptor such as PCnet-FAST III. 

Ironically, I run another rhel 8 system with e1000 driver and it still works though it is not recommended to run on production systems. See RHEL8: Supported virtualization network drivers for more details. 

Most companies use VMware eSXI as their hypervisor and hence by default will be using vmxnet3. So this issue won't occur in production systems. 

2. Grub core upgrade

If using legacy BIOS, grub core wont be upgraded. With UEFI systems, the grub core will be upgraded automatically

3. Red Hat Signed Packages

Below packages not signed by Red Hat and might be removed during upgrade.

convert2rhel - epel-release - gpg-pubkey - libdtrace-ctf - uname26

You can verify and if anything is needed post install, you can install them again if removed during upgrade process.

During the upgrade process, the below error occurred 

...
...
...
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

STDERR:
Warning: Packages marked by Leapp for install not found in repositories metadata: python3-nss jackson-jaxrs-providers uom-lib-javadoc si-units-javadoc slf4j-log4j12 pcp-parfait-agent jackson-annotations jackson-jaxrs-providers-datatypes jackson-core slf4j-ext ldns-utils slf4j-jcl rpcgen python3-pyxattr gdk-pixbuf2-xlib slf4j-jdk14 ivy-local unit-api-javadoc parfait-javadoc jackson-jaxrs-json-provider log4j-over-slf4j jul-to-slf4j uom-se-javadoc log4j12 jackson-module-jaxb-annotations libnsl2-devel parfait-examples uom-systems-javadoc jackson-databind rpcsvc-proto-devel python3-javapackages
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/bin/uname26 from install of util-linux-2.32.1-22.el8.x86_64 conflicts with file from package uname26-1.0-1.el7.x86_64

To overcome this error, I had to manually uninstall uname26 package from the system and start the upgrade again. 

[root@linux75 ~]# rpm -qa|grep uname
uname26-1.0-1.el7.x86_64
[root@linux75 ~]# rpm -e uname26-1.0-1.el7.x86_64

4. Python version

One of the Low Risk Factor provided is that python is found where it is not supported in RHEL8. Install new version if needed. Remediation plan provided is to run the below post upgrade.

alternatives --set python /usr/bin/python3

5. Remote login

This is one of the inhibor reported which needs the below change

PermitRootLogin yes in file /etc/ssh/sshd_config

6. Missing answer in answer file

This is the third inhibitor which I faced requiring the leapp answer confirmation either via command line as below 
# leapp answer --section remove_pam_pkcs11_module_check.confirm=True

or using the web console. We can either add to Remediation Plan and run all at once or run the remediation one by one. In our case, there is only one remediation plan that can be executed via console. So we can click on the Run Remediation button and the the command will be executed as shown below

Once all the inhibitors are remediated, we can run the "leapp preupgrade" again to check whether the system is ready for upgrade. The below screenshot is for reference after inhibitors are eliminated. 

We can see there are High Risk Factors but no inhibitors meaning the upgrade can be done. 

Leapp preupgrade screen will complete as shown below. 

Additional tip: 

If you are running the system with network interfaces by default kernel names such as eth0, eth1 etc,. the following Risk Factor will be raised and it is an inhibitor

This is one of the limitation that I have mentioned in point 5 above. So to fix this, we need to rename the network interfaces which would require downtime as mentioned in this Red Hat solution document. 

# nmcli connection modify "eth0" connection.interface-name "myeth0"

# nmcli connection modify "eth1" connection.interface-name "myeth1"

# reboot

We can proceed with the "leapp upgrade" now after remediating all inhibitors and understanding all Risk Factors involved. 

If the system is not upgradeable, leapp utility will close with the reason explained in /var/log/leapp/leapp-report.txt. 

If the system is upgradeable, leapp utility will download all packages required and the following screen will be seen requiring for reboot

Once reboot is issues, system boots with RHEL 8 initramfs and upgrades the system before rebooting to RHEL 8 system. Once the reboot has been issued, it took around 12 ~ 14 minutes for the upgrade to happen and I'm able to login to RHEL 8.2 server. 

Once the upgrade is successful, we can now continue with the post upgrade tasks such as verifying the post upgrade state of the RHEL 8 system, Performing post upgrade tasks and Applying security policies as described in Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Once all the post tasks are completed, we are now good to test whether the Oracle database starts up without any issues. 

So, I have performed post tasks including the yum update to update system to 8.3. The database comes up without any issues. 

[oracle@linux75 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 25 07:50:38 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1090519040 bytes
Fixed Size                  8791816 bytes
Variable Size             419432696 bytes
Database Buffers          654311424 bytes
Redo Buffers                7983104 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@linux75 ~]$ 
[oracle@linux75 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 25-FEB-2021 07:58:43

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux75.selvapc.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                25-FEB-2021 07:51:04
Uptime                    0 days 0 hr. 7 min. 38 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/db/122/network/admin/listener.ora
Listener Log File         /u01/db/diag/tnslsnr/linux75/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=linux75.selvapc.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=linux75.selvapc.com)(PORT=5500))(Security=(my_wallet_directory=/u01/db/admin/orcl/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "bc06df3df7ad51e2e0538338a8c05a87.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orcl.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclpdb.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@linux75 ~]$

Now to conclude, if you ask me whether it's worth performing the upgrade of the system from RHEL 7 to RHEL 8, I would say we can go ahead with taking in to consideration all the caution and risk factors and remediating them and with proper planning the upgrade would be smoother without any hassles. 

References: 

Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8 | Red Hat Customer Portal

RHEL8: Supported virtualization network drivers - Red Hat Customer Portal

How to assign a specific network interface name in RHEL7, 8 - Red Hat Customer Portal

Happy upgrading!!! 

Converting Oracle Linux 7 to RedHat Enterprise Linux 7

RHEL 8 is available now and Oracle has already started certifying their products on RHEL 8. RHEL 7 (7.7) EUS is going to end by August 2021. Companies are slowly planning to upgrade their on-prem environments to RHEL 8 to comply with the standards. 

I have a system which is running Oracle Linux 7.5. I would like to try upgrading the system to RHEL 8 to see whether is it worth performing an in place upgrade of the OS. 

We will be using subscription-manager and will directly connect to customer portal as I don't have RedHat Satellite installed. Free non-prod/single user subscription can be obtained by registering in https://developers.redhat.com 

Note: There is an in-place upgrade strategy from RHEL 6 to RHEL 7 but is not supported by Redhat but RHEL 7 to RHEL 8 in-place upgrade is officially supported by Redhat via Leapp. So to perform this upgrade, I would like to convert my Oracle Linux to Redhat Linux. In this post, we will look at the conversion process of the same. 

Current environment that I'm using: 

[root@linux75 ~]# uname -a
Linux linux75.selvapc.com 4.1.12-112.16.4.el7uek.x86_64 #2 SMP Mon Mar 12 23:57:12 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@linux75 ~]# cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="7.5"
ID="ol"
VERSION_ID="7.5"
PRETTY_NAME="Oracle Linux Server 7.5"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:5:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.5
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.5
[root@linux75 ~]#

[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 18 00:11:07 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select * from v$version;

BANNER                                                                               CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production              0
PL/SQL Release 12.2.0.1.0 - Production                                                    0
CORE    12.2.0.1.0      Production                                                                0
TNS for Linux: Version 12.2.0.1.0 - Production                                            0
NLSRTL Version 12.2.0.1.0 - Production                                                    0

SQL>

As you can see I'm running Oracle database on Oracle Linux and the whole idea is to check whether the database is still operational without any issues post conversion. 

Step 1: Install convert2rhel package

convert2rhel package won't be available in default repos. We need to add EPEL repo and then install the package.

Results truncated for easy understanding.. 

[root@linux75 ~]#
[root@linux75 ~]#
[root@linux75 ~]# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Loaded plugins: langpacks, ulninfo
epel-release-latest-7.noarch.rpm                                                                                                                                  |  15 kB  00:00:00
Examining /var/tmp/yum-root-94f61j/epel-release-latest-7.noarch.rpm: epel-release-7-13.noarch
Marking /var/tmp/yum-root-94f61j/epel-release-latest-7.noarch.rpm to be installed
Resolving Dependencies
...
...
...

Installed:
  epel-release.noarch 0:7-13

Complete!
[root@linux75 ~]# yum install convert2rhel
Loaded plugins: langpacks, ulninfo
epel/x86_64/metalink                                                                                                                                              | 4.7 kB  00:00:00
epel                                                                                                                                                              | 4.7 kB  00:00:00
(1/3): epel/x86_64/group_gz                                                                                                                                       |  95 kB  00:00:01
(2/3): epel/x86_64/updateinfo                                                                                                                                     | 1.0 MB  00:00:03
(3/3): epel/x86_64/primary_db                                                                                                                                     | 6.9 MB  00:00:07
Resolving Dependencies
...
...
...
  Verifying  : pexpect-2.3-11.el7.noarch                                                                                                                                             2/3
  Verifying  : convert2rhel-0.13-1.el7.noarch                                                                                                                                        3/3

Installed:
  convert2rhel.noarch 0:0.13-1.el7

Dependency Installed:
  pexpect.noarch 0:2.3-11.el7                                                             python-dateutil.noarch 0:1.5-7.el7

Complete!
[root@linux75 ~]#

Step 2: Install Red Hat GPG signing key

If you have a Redhat 7 installation media, then you can mount the DVD and use the below command 

# mount /dev/sr0 /dvd

# rpm --import /dvd/RPM-GPG-KEY-redhat-release

Or if you don't have an installation media handy, you can download the file from Redhat portal Product signing keys and copy the same under /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release.

[root@linux75 ~]# cp /media/sf_Oracle/software/RPM-GPG-KEY-redhat-release.txt /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[root@linux75 ~]#

Step 3: Install Redhat Subscription Management related packages. 

convert2rhel package expects the following packages in the /usr/share/convert2rhel/subscription-manager directory. Download the following packages from Redhat Packages which requires subscription or can be obtained from RHEL 7 installation media

subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm

subscription-manager-1.24.45-1.el7_9.x86_64.rpm

subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm

python-syspurpose-1.24.45-1.el7_9.x86_64.rpm -- needed as a dependency

We will create the directory, copy the packages and also install the packages as sometime the package installation fails via convert2rhel tool due to dependencies. 

[root@linux75 software]# mkdir /usr/share/convert2rhel/subscription-manager
[root@linux75 software]# ls -lrt subscription-manager-*
-rwxrwx---. 1 root vboxsf  340028 Feb 17 16:04 subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm
-rwxrwx---. 1 root vboxsf 1132252 Feb 17 16:27 subscription-manager-1.24.45-1.el7_9.x86_64.rpm
-rwxrwx---. 1 root vboxsf  246792 Feb 17 16:28 subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm
[root@linux75 software]# ls -lrt python-syspurpose*
-rwxrwx---. 1 root vboxsf 280436 Feb 17 16:36 python-syspurpose-1.24.45-1.el7_9.x86_64.rpm
[root@linux75 software]# cp subscription-manager-* /usr/share/convert2rhel/subscription-manager/
[root@linux75 software]#
[root@linux75 software]# rpm -ivh python-syspurpose-1.24.45-1.el7_9.x86_64.rpm
warning: python-syspurpose-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:python-syspurpose-1.24.45-1.el7_9################################# [100%]
[root@linux75 software]# rpm -e rhn-client-tools-2.0.2-21.0.3.el7.noarch --nodeps
[root@linux75 software]# rpm -ivh subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-rhsm-certifi################################# [100%]
[root@linux75 software]# rpm -ivh subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-rhsm-1.24.45################################# [100%]
[root@linux75 software]# rpm -ivh subscription-manager-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-1.24.45-1.el################################# [100%]
[root@linux75 software]#

Step 4: Install redhat-release-server package

convert2rhel package requires redhat-release-server package to register the system to either Satellite or subscription account. So we create /usr/share/convert2rhel/redhat-release/Server/ directory and place the redhat-release-server-7.9-5.el7_9.x86_64.rpm which can be downloaded from Redhat portal. 

[root@linux75 software]# mkdir -p /usr/share/convert2rhel/redhat-release/Server/
[root@linux75 software]# cp redhat-release-server-7.9-5.el7_9.x86_64.rpm /usr/share/convert2rhel/redhat-release/Server/
[root@linux75 software]#

Note: All the packages and release server installed are of version 7.9 and hence when converting Oracle Linux to Redhat Enterprise Linux, the version will be upgraded to latest version 7.9. 

Step 5: Start the convert2rhel utility

We are now prepared for converting our system to RHEL. 
If the database is running, stop the database and other applications. 
Run yum clean all to start yum from clean state and start the convert2rhel utility. 
This step will take more time depending on the system resources and internet bandwidth that you have. My system took around 30 minutes to convert. 

I'm using my Developer subscription account to register my system to RHSM and you may use yours. If you are using Redhat satellite, the utility command might differ to register to Satellite. 

1. Start the utility with RHSM username and password
2. Accept EULA
3. Accept system conversion
4. Accept system conversion
5. Enter number of chosen subscription
6. Accept final system conversion dialog post which automatic rollback is not possible. 
7. Finally reboot

[root@linux75 software]# convert2rhel --username nagulan.selvakumar --password ******
...
...
WARNING - By continuing you accept this EULA.

Continue with the system conversion? [y/n]:y


[02/18/2021 01:57:58] TASK - [Prepare: Gather system information] *******************************
Name:                Oracle Linux Server
OS major version:    7
Architecture:        x86_64
Config filename:     oracle-7-x86_64.cfg
Running the 'rpm -Va' command which can take several minutes. It can be disabled by using the --no-rpm-va option.
...
...
...
rhn-setup-gnome-2.0.2-21.0.3.el7.noarch     N/A       anaconda
rhnsd-5.0.13-10.0.1.el7.x86_64              N/A       anaconda
yum-rhn-plugin-2.0.1-10.0.1.el7.noarch      N/A       anaconda


Continue with the system conversion? [y/n]:y
...
...
...
[02/18/2021 02:00:44] TASK - [Convert: List third-party packages] *******************************
WARNING - Only packages signed by Oracle Linux Server are to be reinstalled. Red Hat support won't be provided for the following third party packages:

Package                         Packager        Repository
-------                         --------        ----------
epel-release-7-13.noarch        Fedora Project  N/A
convert2rhel-0.13-1.el7.noarch  Fedora Project  epel


Continue with the system conversion? [y/n]:y
...
...
...
[02/18/2021 02:01:57] TASK - [Convert: Subscription Manager - Subscribe system] *****************
Building subscription-manager command ...
    ... activation key not found, username and password required
    ... username detected
    ... password detected
Registering system by running subscription-manager command ...

...
...
...
Service Level:       Self-Support
Usage:
Add-ons:
Subscription Type:   Standard
Starts:              01/31/21
Ends:                01/31/22
Entitlement Type:    Physical


Enter number of the chosen subscription:1

Attaching subscription with pool ID 8a85f99a774ac5420177570d75625d4e to the system ...

...
...
...
[02/18/2021 02:06:55] TASK - [Convert: Subscription Manager - Rename repositories] **************
Renamed: /etc/yum.repos.d/epel-testing.repo -> /etc/yum.repos.d/epel-testing.repo.convert2rhel_renamed
Renamed: /etc/yum.repos.d/epel.repo -> /etc/yum.repos.d/epel.repo.convert2rhel_renamed
WARNING - The tool allows rollback of any action until this point.
WARNING - By continuing all further changes on the system will need to be reverted manually by the user, if necessary.

Continue with the system conversion? [y/n]: y



[02/18/2021 02:08:38] TASK - [Convert: Import Red Hat GPG keys] *********************************

[02/18/2021 02:08:38] TASK - [Convert: Prepare kernel] ******************************************
Installing RHEL kernel ...
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
              : manager
...
...
...

[02/18/2021 02:35:32] TASK - [Final: Non-interactive mode] **************************************
For the non-interactive use of the tool, run the following command:
convert2rhel -u nagulan.selvakumar -p ***** --pool 8a85f99a774ac5420177570d75625d4e -v Server -y


WARNING - In order to boot the RHEL kernel, restart of the system is needed.
[root@linux75 software]# reboot

Step 6: Verification

Now we can verify whether system is converted and Oracle database is working without any issues. 

[root@linux75 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
[root@linux75 ~]#
[root@linux75 ~]# subscription-manager list
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.9
Arch:           x86_64
Status:         Subscribed
Status Details:
Starts:         01/31/2021
Ends:           01/31/2022

[root@linux75 ~]#
[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 18 02:45:10 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8628936 bytes
Variable Size             448791864 bytes
Database Buffers          608174080 bytes
Redo Buffers                8146944 bytes
Database mounted.
Database opened.
SQL> select INSTANCE_NAME, HOST_NAME, VERSION from v$instance;

INSTANCE_NAME    HOST_NAME                      VERSION
---------------- ------------------------------ -----------------
orcl             linux75.selvapc.com            12.2.0.1.0

SQL>

All is good....!!! 

References: 

How to convert from CentOS or Oracle Linux to RHEL

Converting from CentOS to RHEL with Convert2RHEL and Satellite

Red Hat Enterprise Linux Life Cycle - Red Hat Customer Portal

Happy Converting !!! :) 

Oracle Grid/RDBMS 19c on RHEL 8 - Tips, issues and work arounds

I did an install of Oracle GI and Database version 19c on RHEL 8 in my home lab (we can download a copy of RHEL8 for personal/non-production use if registered in https://developers.redhat.com) where I encountered a few issues. 

Let's see them with their work arounds in this post. 

1.    Oracle software installation prerequisite steps can be automatically done by installing Database pre-install rpms from the below repository links

http://public-yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm

http://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm

2.    Mandatory patch requirement

RHEL was released later than Oracle 19c. So when attempting to install Oracle 19c software, we need to apply patch along with the installation of the software. The minimum required patch level to apply is GIRU19.6 or DBRU19.6 with few oneOff patches or GIRU19.7 or DBRU19.7 patch. As I'm installing now, post GIRU19.10 is released, I'm patching to the latest available patch. So I'm using the command as below to invoke the Grid setup. 

[oracle@linux-8 19]$ ./gridSetup.sh -applyRU /media/sf_Oracle/software/32226239

oracle@linux-8 db19]$ ./runInstaller -applyRU /media/sf_Oracle/software/32226239/32218454

Patch 32226239: GI RELEASE UPDATE 19.10.0.0.0 can be downloaded from Oracle support. This patch includes DB patch (32218454) as well.

3.    INS-08101 Unexpected error 'supportedOSCheck'

As the software is released prior to RHEL/OL 8, the software doesn't recognizes the OS version and throws the below error. 

Work Around: 

Set the environment variable CV_ASSUME_DISTID to OL7 and then start the Grid setup/installer. 

[oracle@linux-8 19]$ export CV_ASSUME_DISTID=OL7

4.    compat-libcap1-1.10 package missing error

Prerequisites check page shows that compat-libcap1-1.10 package is missing.

Work Around: 

This error can safely be ignored. Fix all other errors apart from this package error and click on Ignore all check box and proceed with the installation. 

5.    OPatch requirement

As we need to apply patch as well, we have to use the latest version of OPatch while installing software along with patching. Check for Patch 6880880 in Oracle support and download the latest patch available with correct version of the OS selected. At the time of writing this post, the latest version is OPatch Version: 12.2.0.1.23

6. ASM FD is not supported

During "Create ASM Disk Group" page, when Configure ASM Filter Driver is selected, the following error is thrown. 

AFD-9213: Configuration file 'symvers-4.18.0-240.10.1.el8_3.x86_64.gz' in the /boot directory does not exist or cannot be read.

Note: If you don't apply RU along when starting the gridSetup, then the error will be as below

AFD-620: AFD is not supported on this operating system version: 'unknown'

Work Around: 

Referring to RedHat support document 5174441, the location of /boot/symvers-$(uname -r).gz has been changed to /usr/lib/modules/$(uname -r)/symvers.gz in RHEL 8.

Also, we are running the below kernel level which is not supported by Oracle for ACFS and ASM FD at the time of installation (up to 193 is supported). Refer Support note 1369107.1 for compatibility check

[root@linux-8 modules]# uname -r        

4.18.0-240.10.1.el8_3.x86_64                

As the patch level is still not officially supported, I don't want to copy the symvers file to /boot and try which might mess up other things. So deselecting the ASM FD for now and continuing installation

7.    Automatic root script execution

If you would like the installer to run the root scripts automatically by providing the information such as root password or sudo privileges, edit the /etc/hosts file with the server details. Else the installer will fail stating unknown host. 

References: 

Requirements for Installing Oracle Database/Client 19c on OL8 or RHEL8 64-bit (x86-64) (Doc ID 2668780.1)

Can't find /boot/symvers-KERNEL_VERSION.gz on RHEL 8 ?

ACFS and AFD 19c Supported Platforms

How To Download And Install The Latest OPatch(6880880) Version (Doc ID 274526.1)

Known Issues and Bugs for Oracle Linux 8 and Red Hat Enterprise Linux 8

Patch 32226239: GI RELEASE UPDATE 19.10.0.0.0

Happy Installing...!!!

Unix - Bits and Pieces Part 1

In this blog post series, I'll be updating the post with random unix commands that might be of use to DBAs at regular intervals.

1. Trim Alert log contents

It becomes tedious job when we have to deal with a huge alert log and it gets worse if we have to dig a month old or a week old alert log contents for troubleshooting. So the below command can be used to trim the contents of alert log for a particular period of time.

# This is the format used
sed -n '/From date pattern / , /To date pattern/p' < input_file > output_file
# This is an example
sed -n '/Jul 23 / , /Jul 26/p' < alter_ORACLESID.log > 4days_trimmed.txt

2. Details of CPU available

The details of memory information is available below
Memory Information of different OS platforms
The Details of CPU available can be found by using the commands below in Linux servers.

# -- gives number of CPU
nproc
# Details of CPU from /proc/cpuinfo in human readable format
lscpu
# Details from cpuinfo directly
cat /proc/cpuinfo

For HP-UX the below commands can be used

# For details of procs
ioscan -fnk|grep proc 
# For number of  procs
ioscan -k|grep processor|wc -l
# For details of overall system 
top

For AIX use any of the below.

prtconf -s
pmcycles -m
lsdev -Cc processor
bindprocessor -q

3. Create multiple directories under all directories

If we are required to create multiple directories inside all the available directories the below command can be used
For eg., lets take a directory where envmt has 4 directories dir1, dir2, dir3, dir4
Now we need to create control,bad,param,log directories under each of all the directories under envmt. Illustration is as follows

$ cd envmt
$ ls -lrt
total 16
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:07 dir4
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:07 dir3
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:07 dir2
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:07 dir1
$ ls -l dir1
total 0
$ ls -l dir2
total 0
$ for dir in */; do mkdir -- "$dir"/{control,bad,param,log}; done
$ ls -l dir1
total 16
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 bad
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 control
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 log
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 param
$ ls -l dir2
total 16
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 bad
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 control
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 log
drwxr-xr-x 2 oracle dba 4096 Jul 27 17:08 param

Now control,bad,param,log directories are created in dir1, dir2, dir3, dir4 directories. This saves a lot of time while doing any job.

4. Search and replace in all files

The following command can be used to search and replace a word in all the files inside a directory. This is useful when we have configuration files stored where all the files should be updated with same username or servername, etc. 

# This is the format used
grep -rl 'search_string' ./ | xargs sed -i 's/search_string/replace_with_string/g'
# This is an example
grep -rl 'STDBCS01' ./ | xargs sed -i 's/STDBCS01/DVDBCS01/g'

If this find and replace is required for a single file, then it can be done by opening the file in vi editor

# open file
vi file_name
# Use below command to find and replace 
# g denotes global. Remove g if find and replace is required only for the particular line
:%s/search_string/replace_with_string/g
# save file and exit
:wq

5. Email attachment from servers

Email attachments can be sent with uuencode and the command is as below.
Tested in AIX and Linux

uuencode file1 file1 |mailx -s "Subject" nagulan.selvakumar@my.company.com

The below command is used to send multiple attachments in a single email

uuencode r1.tar.gz r1.tar.gz > /tmp/out.mail
uuencode r2.tar.gz r2.tar.gz >> /tmp/out.mail
uuencode r3.tar.gz r3.tar.gz >> /tmp/out.mail
echo "msg body"  >> /tmp/out.mail
mail -s "Reports" nagulan.selvakumar@my.company.com < /tmp/out.mail

We do have another package "mutt" to send emails which can be utilized as below.
Tested in Linux

echo "This is the message body" | mutt -a file_to_attach -s "subject of message" nagulan.selvakumar@my.company.com 

Happy working!

SCP without password prompt

Many a times, we are in need of copying files from one server to another for our work to continue. SCP is a powerful unix utility to do file copy in a secure way. Each time when you do an scp, it prompts for the password of the target server's user. This is very good in terms of security, but might be annoying when you do a large set of file copy.

This password prompt would halt your work by some means. For e.g., you might had a script configured to pass files from source server to target server where scp does the file copy. Here if you want to pass the password each and everytime, the automated script is of no mean.

Let's see how to do scp without a password prompt.

This is to do scp without password in scripts.

First step is to create a key pair between the servers.

The syntax to create the key pair is:

$ ssh-keygen -t rsa

In response, you should see:

Generating public/private rsa key pair

Enter file in which to save the key ...

Press Enter to accept this.

In response, you should see:

Enter passphrase (empty for no passphrase):

You don't need a passphrase, so press Enter twice.

In response, you should see:

Your identification has been saved in ...

Your public key has been saved in ...

Note the name and location of the public key just generated. It always ends in .pub.

Below is an example from RHEL 8 box

[oracle@linux-8 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:1qKWyoZ9GN0jpOz0ikOv+dkQk9bVGY8PxcDfjp0Ho7Y oracle@linux-8.selvapc.com
The key's randomart image is:
+---[RSA 3072]----+
|         .oo.    |
|         ..*.    |
|        . =...   |
|     o.. . o. +  |
|   .=+..S . .= + |
|  ..=oo+o.  + + .|
| . *.++. . . . . |
|  o+**o     E    |
|  +=*o.          |
+----[SHA256]-----+
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ ls -lrt ~/.ssh/
total 12
-rw-r--r--. 1 oracle oinstall  699 Mar 10 00:23 known_hosts
-rw-------. 1 oracle oinstall 2622 Mar 11 01:49 id_rsa
-rw-r--r--. 1 oracle oinstall  580 Mar 11 01:49 id_rsa.pub
[oracle@linux-8 ~]$

Copy the public key just generated to all of your remote Linux boxes. You can use scp or FTP or whatever to make the copy. 

Assuming you're using root, the key must be contained in the file /root/.ssh/authorized_keys. 

Or, if you are logging in as a user, for example, oracle, it would be in ~/.ssh/authorized_keys. Notice that the authorized_keys file can contain keys from other PCs. So, if the file already exists and contains text, you need to append the contents of your public key file to what already is there.

On remote machine.. 

$ cat id_rsa.pub >>~/.ssh/authorized_keys

$ chmod 700 ~/.ssh/authorized_keys

With the above procedure completely done, you can copy your files without the prompt for the password.

Update: 11 Mar 2021

Added o/p of ssh-keygen from a RHEL 8 box