Looking for something? Try here..

Thursday, February 25, 2021

RHEL 7 to RHEL 8 upgrade on an Oracle RDBMS installed server

 In my last post, I've converted my Oracle Linux to Red Hat Enterprise Linux and we saw that the database was running without any issues post conversion as well. The main reason I did the conversion is to learn a few things from Linux server administration and also I need a RHEL 7 server hosting Oracle database to test the upgrade from RHEL 7 to RHEL 8. This upgrade is officially supported by Red Hat. 

In this post I'll be following exactly what is provided in the below official Red Hat link which is a straight forward step by step procedure where I'll discuss what all things needs to be taken care and how the upgrade goes and importantly to test the functionality of Oracle database running on the upgraded server post upgrade. 

The step by step procedure is outlined in the below link 

Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Unlike the previous RHEL 6 to RHEL 7 upgrade document (though not officially supported), this document usage is very easy to follow and there seems to be no hiccups anywhere even for a beginner in System Administration. So let's get to our observation and results. 

1. You need to be on latest available version, currently 7.9 

2. Third party apps can also be migrated to upgraded version using custom actors. (I've not done this as I don't have any)

3. During upgrade process, Leapp utility sets SELinux mode to permissive post which user needs to change it to enforcing. 

4. There is a slightly different method of upgrade if you run SAP HANA (we are not discussing here)

5. There are a few limitations as well. So understand these before planning for the upgrade. 

6. If you don't have a satellite server or Red Hat subscription, you can also setup custom repo containing at least latest rpms of rhel 7 server, rhel 7 server extras, rhel 8 base OS and rhel 8 appstream

7. Install leapp utility along with leapp-repository and cockpit-leapp 

The main aspect of this upgrade is the pre upgrade report provided by the leapp utility. 
Run the "leapp preupgrade" command either from a terminal or from the server web console to identify potential problems associated with the upgrade and their remediation methods for a smoother upgrade process. 

For Linux admin beginners (like me), web console is a very attractive, easy to use console accessible via web using the https://localhost:9090 or https:(servername)or(ip):9090 with root or user with sudo permissions. 

Sample window looks like the below

In my system, I've ran the preupgrade and below are the observations. Once after running the leapp preupgrade command, we can click on the "Upgrade Report" on the left hand side of the console to view the report with its severity or Risk factor, information messages, remediation plans and inhibitors.

Sample report of my test system


We can see there are 6 Titles reported as High Risk factor in which 3 are Inhibitors. Inhibitors needs to be remediated and if not, upgrade will not progress. Let's take a look at each of them. 

1. Disable e1000 driver

First inhibitor in the report is the unsupported driver found in the system e1000. e1000 is the network driver installed when choosing Intel PRO/1000 MT desktop as network adaptor while configuring the VM on VirtualBox. Simple fix is to use a different adaptor such as PCnet-FAST III. 


Ironically, I run another rhel 8 system with e1000 driver and it still works though it is not recommended to run on production systems. See RHEL8: Supported virtualization network drivers for more details. 
Most companies use VMware eSXI as their hypervisor and hence by default will be using vmxnet3. So this issue won't occur in production systems. 

2. Grub core upgrade

If using legacy BIOS, grub core wont be upgraded. With UEFI systems, the grub core will be upgraded automatically

3. Red Hat Signed Packages

Below packages not signed by Red Hat and might be removed during upgrade.

convert2rhel - epel-release - gpg-pubkey - libdtrace-ctf - uname26

You can verify and if anything is needed post install, you can install them again if removed during upgrade process.
During the upgrade process, the below error occurred 
...
...
...
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

STDERR:
Warning: Packages marked by Leapp for install not found in repositories metadata: python3-nss jackson-jaxrs-providers uom-lib-javadoc si-units-javadoc slf4j-log4j12 pcp-parfait-agent jackson-annotations jackson-jaxrs-providers-datatypes jackson-core slf4j-ext ldns-utils slf4j-jcl rpcgen python3-pyxattr gdk-pixbuf2-xlib slf4j-jdk14 ivy-local unit-api-javadoc parfait-javadoc jackson-jaxrs-json-provider log4j-over-slf4j jul-to-slf4j uom-se-javadoc log4j12 jackson-module-jaxb-annotations libnsl2-devel parfait-examples uom-systems-javadoc jackson-databind rpcsvc-proto-devel python3-javapackages
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/bin/uname26 from install of util-linux-2.32.1-22.el8.x86_64 conflicts with file from package uname26-1.0-1.el7.x86_64
To overcome this error, I had to manually uninstall uname26 package from the system and start the upgrade again. 
[root@linux75 ~]# rpm -qa|grep uname
uname26-1.0-1.el7.x86_64
[root@linux75 ~]# rpm -e uname26-1.0-1.el7.x86_64

4. Python version

One of the Low Risk Factor provided is that python is found where it is not supported in RHEL8. Install new version if needed. Remediation plan provided is to run the below post upgrade.

alternatives --set python /usr/bin/python3

5. Remote login

This is one of the inhibor reported which needs the below change

PermitRootLogin yes in file /etc/ssh/sshd_config

6. Missing answer in answer file

This is the third inhibitor which I faced requiring the leapp answer confirmation either via command line as below 
# leapp answer --section remove_pam_pkcs11_module_check.confirm=True

or using the web console. We can either add to Remediation Plan and run all at once or run the remediation one by one. In our case, there is only one remediation plan that can be executed via console. So we can click on the Run Remediation button and the the command will be executed as shown below


Once all the inhibitors are remediated, we can run the "leapp preupgrade" again to check whether the system is ready for upgrade. The below screenshot is for reference after inhibitors are eliminated. 


We can see there are High Risk Factors but no inhibitors meaning the upgrade can be done. 
Leapp preupgrade screen will complete as shown below. 


Additional tip: 

If you are running the system with network interfaces by default kernel names such as eth0, eth1 etc,. the following Risk Factor will be raised and it is an inhibitor

This is one of the limitation that I have mentioned in point 5 above. So to fix this, we need to rename the network interfaces which would require downtime as mentioned in this Red Hat solution document

# nmcli connection modify "eth0" connection.interface-name "myeth0"
# nmcli connection modify "eth1" connection.interface-name "myeth1"
# reboot

We can proceed with the "leapp upgrade" now after remediating all inhibitors and understanding all Risk Factors involved. 
If the system is not upgradeable, leapp utility will close with the reason explained in /var/log/leapp/leapp-report.txt. 
If the system is upgradeable, leapp utility will download all packages required and the following screen will be seen requiring for reboot


Once reboot is issues, system boots with RHEL 8 initramfs and upgrades the system before rebooting to RHEL 8 system. Once the reboot has been issued, it took around 12 ~ 14 minutes for the upgrade to happen and I'm able to login to RHEL 8.2 server. 


Once the upgrade is successful, we can now continue with the post upgrade tasks such as verifying the post upgrade state of the RHEL 8 system, Performing post upgrade tasks and Applying security policies as described in Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Once all the post tasks are completed, we are now good to test whether the Oracle database starts up without any issues. 

So, I have performed post tasks including the yum update to update system to 8.3. The database comes up without any issues. 
[oracle@linux75 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 25 07:50:38 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1090519040 bytes
Fixed Size                  8791816 bytes
Variable Size             419432696 bytes
Database Buffers          654311424 bytes
Redo Buffers                7983104 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@linux75 ~]$ 
[oracle@linux75 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 25-FEB-2021 07:58:43

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux75.selvapc.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                25-FEB-2021 07:51:04
Uptime                    0 days 0 hr. 7 min. 38 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/db/122/network/admin/listener.ora
Listener Log File         /u01/db/diag/tnslsnr/linux75/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=linux75.selvapc.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=linux75.selvapc.com)(PORT=5500))(Security=(my_wallet_directory=/u01/db/admin/orcl/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "bc06df3df7ad51e2e0538338a8c05a87.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orcl.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclpdb.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@linux75 ~]$


Now to conclude, if you ask me whether it's worth performing the upgrade of the system from RHEL 7 to RHEL 8, I would say we can go ahead with taking in to consideration all the caution and risk factors and remediating them and with proper planning the upgrade would be smoother without any hassles. 

References: 


Happy upgrading!!! 

2 comments:

  1. What others could be the factors to migrate from Oracle Linux to Red Hat? One of the great things about OL is that we don’t need a subscription for update the systems.

    ReplyDelete
    Replies
    1. Hi, If you are already on OL, then it's up to you to decide whether to move to RHEL. RHEL subscription is needed so that the packages can't be tampered as they are supporting it. Also subscription is needed only if want the errata and updates to be applied.
      OL is basically tuned to run Oracle applications whereas RHEL is a complete Linux where you can also run UEK to support Oracle applications if needed.
      Mostly companies use RHEL as their standard OS when it comes to production use but we can use OL as well. Thanks!

      Delete