Looking for something? Try here..

Thursday, February 25, 2021

RHEL 7 to RHEL 8 upgrade on an Oracle RDBMS installed server

 In my last post, I've converted my Oracle Linux to Red Hat Enterprise Linux and we saw that the database was running without any issues post conversion as well. The main reason I did the conversion is to learn a few things from Linux server administration and also I need a RHEL 7 server hosting Oracle database to test the upgrade from RHEL 7 to RHEL 8. This upgrade is officially supported by Red Hat. 

In this post I'll be following exactly what is provided in the below official Red Hat link which is a straight forward step by step procedure where I'll discuss what all things needs to be taken care and how the upgrade goes and importantly to test the functionality of Oracle database running on the upgraded server post upgrade. 

The step by step procedure is outlined in the below link 

Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Unlike the previous RHEL 6 to RHEL 7 upgrade document (though not officially supported), this document usage is very easy to follow and there seems to be no hiccups anywhere even for a beginner in System Administration. So let's get to our observation and results. 

1. You need to be on latest available version, currently 7.9 

2. Third party apps can also be migrated to upgraded version using custom actors. (I've not done this as I don't have any)

3. During upgrade process, Leapp utility sets SELinux mode to permissive post which user needs to change it to enforcing. 

4. There is a slightly different method of upgrade if you run SAP HANA (we are not discussing here)

5. There are a few limitations as well. So understand these before planning for the upgrade. 

6. If you don't have a satellite server or Red Hat subscription, you can also setup custom repo containing at least latest rpms of rhel 7 server, rhel 7 server extras, rhel 8 base OS and rhel 8 appstream

7. Install leapp utility along with leapp-repository and cockpit-leapp 

The main aspect of this upgrade is the pre upgrade report provided by the leapp utility. 
Run the "leapp preupgrade" command either from a terminal or from the server web console to identify potential problems associated with the upgrade and their remediation methods for a smoother upgrade process. 

For Linux admin beginners (like me), web console is a very attractive, easy to use console accessible via web using the https://localhost:9090 or https:(servername)or(ip):9090 with root or user with sudo permissions. 

Sample window looks like the below

In my system, I've ran the preupgrade and below are the observations. Once after running the leapp preupgrade command, we can click on the "Upgrade Report" on the left hand side of the console to view the report with its severity or Risk factor, information messages, remediation plans and inhibitors.

Sample report of my test system


We can see there are 6 Titles reported as High Risk factor in which 3 are Inhibitors. Inhibitors needs to be remediated and if not, upgrade will not progress. Let's take a look at each of them. 

1. Disable e1000 driver

First inhibitor in the report is the unsupported driver found in the system e1000. e1000 is the network driver installed when choosing Intel PRO/1000 MT desktop as network adaptor while configuring the VM on VirtualBox. Simple fix is to use a different adaptor such as PCnet-FAST III. 


Ironically, I run another rhel 8 system with e1000 driver and it still works though it is not recommended to run on production systems. See RHEL8: Supported virtualization network drivers for more details. 
Most companies use VMware eSXI as their hypervisor and hence by default will be using vmxnet3. So this issue won't occur in production systems. 

2. Grub core upgrade

If using legacy BIOS, grub core wont be upgraded. With UEFI systems, the grub core will be upgraded automatically

3. Red Hat Signed Packages

Below packages not signed by Red Hat and might be removed during upgrade.

convert2rhel - epel-release - gpg-pubkey - libdtrace-ctf - uname26

You can verify and if anything is needed post install, you can install them again if removed during upgrade process.
During the upgrade process, the below error occurred 
...
...
...
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

STDERR:
Warning: Packages marked by Leapp for install not found in repositories metadata: python3-nss jackson-jaxrs-providers uom-lib-javadoc si-units-javadoc slf4j-log4j12 pcp-parfait-agent jackson-annotations jackson-jaxrs-providers-datatypes jackson-core slf4j-ext ldns-utils slf4j-jcl rpcgen python3-pyxattr gdk-pixbuf2-xlib slf4j-jdk14 ivy-local unit-api-javadoc parfait-javadoc jackson-jaxrs-json-provider log4j-over-slf4j jul-to-slf4j uom-se-javadoc log4j12 jackson-module-jaxb-annotations libnsl2-devel parfait-examples uom-systems-javadoc jackson-databind rpcsvc-proto-devel python3-javapackages
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/bin/uname26 from install of util-linux-2.32.1-22.el8.x86_64 conflicts with file from package uname26-1.0-1.el7.x86_64
To overcome this error, I had to manually uninstall uname26 package from the system and start the upgrade again. 
[root@linux75 ~]# rpm -qa|grep uname
uname26-1.0-1.el7.x86_64
[root@linux75 ~]# rpm -e uname26-1.0-1.el7.x86_64

4. Python version

One of the Low Risk Factor provided is that python is found where it is not supported in RHEL8. Install new version if needed. Remediation plan provided is to run the below post upgrade.

alternatives --set python /usr/bin/python3

5. Remote login

This is one of the inhibor reported which needs the below change

PermitRootLogin yes in file /etc/ssh/sshd_config

6. Missing answer in answer file

This is the third inhibitor which I faced requiring the leapp answer confirmation either via command line as below 
# leapp answer --section remove_pam_pkcs11_module_check.confirm=True

or using the web console. We can either add to Remediation Plan and run all at once or run the remediation one by one. In our case, there is only one remediation plan that can be executed via console. So we can click on the Run Remediation button and the the command will be executed as shown below


Once all the inhibitors are remediated, we can run the "leapp preupgrade" again to check whether the system is ready for upgrade. The below screenshot is for reference after inhibitors are eliminated. 


We can see there are High Risk Factors but no inhibitors meaning the upgrade can be done. 
Leapp preupgrade screen will complete as shown below. 


Additional tip: 

If you are running the system with network interfaces by default kernel names such as eth0, eth1 etc,. the following Risk Factor will be raised and it is an inhibitor

This is one of the limitation that I have mentioned in point 5 above. So to fix this, we need to rename the network interfaces which would require downtime as mentioned in this Red Hat solution document

# nmcli connection modify "eth0" connection.interface-name "myeth0"
# nmcli connection modify "eth1" connection.interface-name "myeth1"
# reboot

We can proceed with the "leapp upgrade" now after remediating all inhibitors and understanding all Risk Factors involved. 
If the system is not upgradeable, leapp utility will close with the reason explained in /var/log/leapp/leapp-report.txt. 
If the system is upgradeable, leapp utility will download all packages required and the following screen will be seen requiring for reboot


Once reboot is issues, system boots with RHEL 8 initramfs and upgrades the system before rebooting to RHEL 8 system. Once the reboot has been issued, it took around 12 ~ 14 minutes for the upgrade to happen and I'm able to login to RHEL 8.2 server. 


Once the upgrade is successful, we can now continue with the post upgrade tasks such as verifying the post upgrade state of the RHEL 8 system, Performing post upgrade tasks and Applying security policies as described in Upgrading from RHEL 7 to RHEL 8 Red Hat Enterprise Linux 8

Once all the post tasks are completed, we are now good to test whether the Oracle database starts up without any issues. 

So, I have performed post tasks including the yum update to update system to 8.3. The database comes up without any issues. 
[oracle@linux75 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 25 07:50:38 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1090519040 bytes
Fixed Size                  8791816 bytes
Variable Size             419432696 bytes
Database Buffers          654311424 bytes
Redo Buffers                7983104 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@linux75 ~]$ 
[oracle@linux75 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 25-FEB-2021 07:58:43

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux75.selvapc.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                25-FEB-2021 07:51:04
Uptime                    0 days 0 hr. 7 min. 38 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/db/122/network/admin/listener.ora
Listener Log File         /u01/db/diag/tnslsnr/linux75/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=linux75.selvapc.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=linux75.selvapc.com)(PORT=5500))(Security=(my_wallet_directory=/u01/db/admin/orcl/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "bc06df3df7ad51e2e0538338a8c05a87.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orcl.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclpdb.selvapc.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@linux75 ~]$


Now to conclude, if you ask me whether it's worth performing the upgrade of the system from RHEL 7 to RHEL 8, I would say we can go ahead with taking in to consideration all the caution and risk factors and remediating them and with proper planning the upgrade would be smoother without any hassles. 

References: 


Happy upgrading!!! 

Thursday, February 18, 2021

Converting Oracle Linux 7 to RedHat Enterprise Linux 7

RHEL 8 is available now and Oracle has already started certifying their products on RHEL 8. RHEL 7 (7.7) EUS is going to end by August 2021. Companies are slowly planning to upgrade their on-prem environments to RHEL 8 to comply with the standards. 

I have a system which is running Oracle Linux 7.5. I would like to try upgrading the system to RHEL 8 to see whether is it worth performing an in place upgrade of the OS. 

We will be using subscription-manager and will directly connect to customer portal as I don't have RedHat Satellite installed. Free non-prod/single user subscription can be obtained by registering in https://developers.redhat.com 

Note: There is an in-place upgrade strategy from RHEL 6 to RHEL 7 but is not supported by Redhat but RHEL 7 to RHEL 8 in-place upgrade is officially supported by Redhat via Leapp. So to perform this upgrade, I would like to convert my Oracle Linux to Redhat Linux. In this post, we will look at the conversion process of the same. 

Current environment that I'm using: 

[root@linux75 ~]# uname -a
Linux linux75.selvapc.com 4.1.12-112.16.4.el7uek.x86_64 #2 SMP Mon Mar 12 23:57:12 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@linux75 ~]# cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="7.5"
ID="ol"
VERSION_ID="7.5"
PRETTY_NAME="Oracle Linux Server 7.5"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:5:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.5
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.5
[root@linux75 ~]#

[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 18 00:11:07 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select * from v$version;

BANNER                                                                               CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production              0
PL/SQL Release 12.2.0.1.0 - Production                                                    0
CORE    12.2.0.1.0      Production                                                                0
TNS for Linux: Version 12.2.0.1.0 - Production                                            0
NLSRTL Version 12.2.0.1.0 - Production                                                    0

SQL>

As you can see I'm running Oracle database on Oracle Linux and the whole idea is to check whether the database is still operational without any issues post conversion. 

Step 1: Install convert2rhel package

convert2rhel package won't be available in default repos. We need to add EPEL repo and then install the package.
Results truncated for easy understanding.. 

[root@linux75 ~]#
[root@linux75 ~]#
[root@linux75 ~]# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Loaded plugins: langpacks, ulninfo
epel-release-latest-7.noarch.rpm                                                                                                                                  |  15 kB  00:00:00
Examining /var/tmp/yum-root-94f61j/epel-release-latest-7.noarch.rpm: epel-release-7-13.noarch
Marking /var/tmp/yum-root-94f61j/epel-release-latest-7.noarch.rpm to be installed
Resolving Dependencies
...
...
...

Installed:
  epel-release.noarch 0:7-13

Complete!
[root@linux75 ~]# yum install convert2rhel
Loaded plugins: langpacks, ulninfo
epel/x86_64/metalink                                                                                                                                              | 4.7 kB  00:00:00
epel                                                                                                                                                              | 4.7 kB  00:00:00
(1/3): epel/x86_64/group_gz                                                                                                                                       |  95 kB  00:00:01
(2/3): epel/x86_64/updateinfo                                                                                                                                     | 1.0 MB  00:00:03
(3/3): epel/x86_64/primary_db                                                                                                                                     | 6.9 MB  00:00:07
Resolving Dependencies
...
...
...
  Verifying  : pexpect-2.3-11.el7.noarch                                                                                                                                             2/3
  Verifying  : convert2rhel-0.13-1.el7.noarch                                                                                                                                        3/3

Installed:
  convert2rhel.noarch 0:0.13-1.el7

Dependency Installed:
  pexpect.noarch 0:2.3-11.el7                                                             python-dateutil.noarch 0:1.5-7.el7

Complete!
[root@linux75 ~]#
Step 2: Install Red Hat GPG signing key

If you have a Redhat 7 installation media, then you can mount the DVD and use the below command 

# mount /dev/sr0 /dvd
# rpm --import /dvd/RPM-GPG-KEY-redhat-release

Or if you don't have an installation media handy, you can download the file from Redhat portal Product signing keys and copy the same under /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release.

[root@linux75 ~]# cp /media/sf_Oracle/software/RPM-GPG-KEY-redhat-release.txt /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[root@linux75 ~]#
Step 3: Install Redhat Subscription Management related packages. 

convert2rhel package expects the following packages in the /usr/share/convert2rhel/subscription-manager directory. Download the following packages from Redhat Packages which requires subscription or can be obtained from RHEL 7 installation media

subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm
subscription-manager-1.24.45-1.el7_9.x86_64.rpm
subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm
python-syspurpose-1.24.45-1.el7_9.x86_64.rpm -- needed as a dependency

We will create the directory, copy the packages and also install the packages as sometime the package installation fails via convert2rhel tool due to dependencies. 
[root@linux75 software]# mkdir /usr/share/convert2rhel/subscription-manager
[root@linux75 software]# ls -lrt subscription-manager-*
-rwxrwx---. 1 root vboxsf  340028 Feb 17 16:04 subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm
-rwxrwx---. 1 root vboxsf 1132252 Feb 17 16:27 subscription-manager-1.24.45-1.el7_9.x86_64.rpm
-rwxrwx---. 1 root vboxsf  246792 Feb 17 16:28 subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm
[root@linux75 software]# ls -lrt python-syspurpose*
-rwxrwx---. 1 root vboxsf 280436 Feb 17 16:36 python-syspurpose-1.24.45-1.el7_9.x86_64.rpm
[root@linux75 software]# cp subscription-manager-* /usr/share/convert2rhel/subscription-manager/
[root@linux75 software]#
[root@linux75 software]# rpm -ivh python-syspurpose-1.24.45-1.el7_9.x86_64.rpm
warning: python-syspurpose-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:python-syspurpose-1.24.45-1.el7_9################################# [100%]
[root@linux75 software]# rpm -e rhn-client-tools-2.0.2-21.0.3.el7.noarch --nodeps
[root@linux75 software]# rpm -ivh subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-rhsm-certificates-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-rhsm-certifi################################# [100%]
[root@linux75 software]# rpm -ivh subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-rhsm-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-rhsm-1.24.45################################# [100%]
[root@linux75 software]# rpm -ivh subscription-manager-1.24.45-1.el7_9.x86_64.rpm
warning: subscription-manager-1.24.45-1.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:subscription-manager-1.24.45-1.el################################# [100%]
[root@linux75 software]#
Step 4: Install redhat-release-server package

convert2rhel package requires redhat-release-server package to register the system to either Satellite or subscription account. So we create /usr/share/convert2rhel/redhat-release/Server/ directory and place the redhat-release-server-7.9-5.el7_9.x86_64.rpm which can be downloaded from Redhat portal. 

[root@linux75 software]# mkdir -p /usr/share/convert2rhel/redhat-release/Server/
[root@linux75 software]# cp redhat-release-server-7.9-5.el7_9.x86_64.rpm /usr/share/convert2rhel/redhat-release/Server/
[root@linux75 software]#
Note: All the packages and release server installed are of version 7.9 and hence when converting Oracle Linux to Redhat Enterprise Linux, the version will be upgraded to latest version 7.9. 

Step 5: Start the convert2rhel utility

We are now prepared for converting our system to RHEL. 
If the database is running, stop the database and other applications. 
Run yum clean all to start yum from clean state and start the convert2rhel utility. 
This step will take more time depending on the system resources and internet bandwidth that you have. My system took around 30 minutes to convert. 

I'm using my Developer subscription account to register my system to RHSM and you may use yours. If you are using Redhat satellite, the utility command might differ to register to Satellite. 

  1. Start the utility with RHSM username and password
  2. Accept EULA
  3. Accept system conversion
  4. Accept system conversion
  5. Enter number of chosen subscription
  6. Accept final system conversion dialog post which automatic rollback is not possible. 
  7. Finally reboot

[root@linux75 software]# convert2rhel --username nagulan.selvakumar --password ******
...
...
WARNING - By continuing you accept this EULA.

Continue with the system conversion? [y/n]:y


[02/18/2021 01:57:58] TASK - [Prepare: Gather system information] *******************************
Name:                Oracle Linux Server
OS major version:    7
Architecture:        x86_64
Config filename:     oracle-7-x86_64.cfg
Running the 'rpm -Va' command which can take several minutes. It can be disabled by using the --no-rpm-va option.
...
...
...
rhn-setup-gnome-2.0.2-21.0.3.el7.noarch     N/A       anaconda
rhnsd-5.0.13-10.0.1.el7.x86_64              N/A       anaconda
yum-rhn-plugin-2.0.1-10.0.1.el7.noarch      N/A       anaconda


Continue with the system conversion? [y/n]:y
...
...
...
[02/18/2021 02:00:44] TASK - [Convert: List third-party packages] *******************************
WARNING - Only packages signed by Oracle Linux Server are to be reinstalled. Red Hat support won't be provided for the following third party packages:

Package                         Packager        Repository
-------                         --------        ----------
epel-release-7-13.noarch        Fedora Project  N/A
convert2rhel-0.13-1.el7.noarch  Fedora Project  epel


Continue with the system conversion? [y/n]:y
...
...
...
[02/18/2021 02:01:57] TASK - [Convert: Subscription Manager - Subscribe system] *****************
Building subscription-manager command ...
    ... activation key not found, username and password required
    ... username detected
    ... password detected
Registering system by running subscription-manager command ...

...
...
...
Service Level:       Self-Support
Usage:
Add-ons:
Subscription Type:   Standard
Starts:              01/31/21
Ends:                01/31/22
Entitlement Type:    Physical


Enter number of the chosen subscription:1

Attaching subscription with pool ID 8a85f99a774ac5420177570d75625d4e to the system ...

...
...
...
[02/18/2021 02:06:55] TASK - [Convert: Subscription Manager - Rename repositories] **************
Renamed: /etc/yum.repos.d/epel-testing.repo -> /etc/yum.repos.d/epel-testing.repo.convert2rhel_renamed
Renamed: /etc/yum.repos.d/epel.repo -> /etc/yum.repos.d/epel.repo.convert2rhel_renamed
WARNING - The tool allows rollback of any action until this point.
WARNING - By continuing all further changes on the system will need to be reverted manually by the user, if necessary.

Continue with the system conversion? [y/n]: y



[02/18/2021 02:08:38] TASK - [Convert: Import Red Hat GPG keys] *********************************

[02/18/2021 02:08:38] TASK - [Convert: Prepare kernel] ******************************************
Installing RHEL kernel ...
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
              : manager
...
...
...

[02/18/2021 02:35:32] TASK - [Final: Non-interactive mode] **************************************
For the non-interactive use of the tool, run the following command:
convert2rhel -u nagulan.selvakumar -p ***** --pool 8a85f99a774ac5420177570d75625d4e -v Server -y


WARNING - In order to boot the RHEL kernel, restart of the system is needed.
[root@linux75 software]# reboot

Step 6: Verification

Now we can verify whether system is converted and Oracle database is working without any issues. 
[root@linux75 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
[root@linux75 ~]#
[root@linux75 ~]# subscription-manager list
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.9
Arch:           x86_64
Status:         Subscribed
Status Details:
Starts:         01/31/2021
Ends:           01/31/2022

[root@linux75 ~]#
[oracle@linux75 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 18 02:45:10 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  8628936 bytes
Variable Size             448791864 bytes
Database Buffers          608174080 bytes
Redo Buffers                8146944 bytes
Database mounted.
Database opened.
SQL> select INSTANCE_NAME, HOST_NAME, VERSION from v$instance;

INSTANCE_NAME    HOST_NAME                      VERSION
---------------- ------------------------------ -----------------
orcl             linux75.selvapc.com            12.2.0.1.0

SQL>
All is good....!!! 

References: 

How to convert from CentOS or Oracle Linux to RHEL

Happy Converting !!! :) 

Thursday, February 11, 2021

Oracle Grid/RDBMS 19c on RHEL 8 - Tips, issues and work arounds

I did an install of Oracle GI and Database version 19c on RHEL 8 in my home lab (we can download a copy of RHEL8 for personal/non-production use if registered in https://developers.redhat.com) where I encountered a few issues. 

Let's see them with their work arounds in this post. 

1.    Oracle software installation prerequisite steps can be automatically done by installing Database pre-install rpms from the below repository links

http://public-yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm

http://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm

2.    Mandatory patch requirement

RHEL was released later than Oracle 19c. So when attempting to install Oracle 19c software, we need to apply patch along with the installation of the software. The minimum required patch level to apply is GIRU19.6 or DBRU19.6 with few oneOff patches or GIRU19.7 or DBRU19.7 patch. As I'm installing now, post GIRU19.10 is released, I'm patching to the latest available patch. So I'm using the command as below to invoke the Grid setup. 

[oracle@linux-8 19]$ ./gridSetup.sh -applyRU /media/sf_Oracle/software/32226239

oracle@linux-8 db19]$ ./runInstaller -applyRU /media/sf_Oracle/software/32226239/32218454

Patch 32226239: GI RELEASE UPDATE 19.10.0.0.0 can be downloaded from Oracle support. This patch includes DB patch (32218454) as well.

3.    INS-08101 Unexpected error 'supportedOSCheck'

As the software is released prior to RHEL/OL 8, the software doesn't recognizes the OS version and throws the below error. 

Work Around: 
Set the environment variable CV_ASSUME_DISTID to OL7 and then start the Grid setup/installer. 

[oracle@linux-8 19]$ export CV_ASSUME_DISTID=OL7

4.    compat-libcap1-1.10 package missing error

Prerequisites check page shows that compat-libcap1-1.10 package is missing.

Work Around: 
This error can safely be ignored. Fix all other errors apart from this package error and click on Ignore all check box and proceed with the installation. 

5.    OPatch requirement

As we need to apply patch as well, we have to use the latest version of OPatch while installing software along with patching. Check for Patch 6880880 in Oracle support and download the latest patch available with correct version of the OS selected. At the time of writing this post, the latest version is OPatch Version: 12.2.0.1.23

6. ASM FD is not supported

During "Create ASM Disk Group" page, when Configure ASM Filter Driver is selected, the following error is thrown. 
AFD-9213: Configuration file 'symvers-4.18.0-240.10.1.el8_3.x86_64.gz' in the /boot directory does not exist or cannot be read.

Note: If you don't apply RU along when starting the gridSetup, then the error will be as below
AFD-620: AFD is not supported on this operating system version: 'unknown'

Work Around: 
Referring to RedHat support document 5174441, the location of /boot/symvers-$(uname -r).gz has been changed to /usr/lib/modules/$(uname -r)/symvers.gz in RHEL 8.
Also, we are running the below kernel level which is not supported by Oracle for ACFS and ASM FD at the time of installation (up to 193 is supported). Refer Support note 1369107.1 for compatibility check

[root@linux-8 modules]# uname -r        
4.18.0-240.10.1.el8_3.x86_64                


As the patch level is still not officially supported, I don't want to copy the symvers file to /boot and try which might mess up other things. So deselecting the ASM FD for now and continuing installation

7.    Automatic root script execution

If you would like the installer to run the root scripts automatically by providing the information such as root password or sudo privileges, edit the /etc/hosts file with the server details. Else the installer will fail stating unknown host. 



References: 
Requirements for Installing Oracle Database/Client 19c on OL8 or RHEL8 64-bit (x86-64) (Doc ID 2668780.1)
ACFS and AFD 19c Supported Platforms
How To Download And Install The Latest OPatch(6880880) Version (Doc ID 274526.1)
Known Issues and Bugs for Oracle Linux 8 and Red Hat Enterprise Linux 8
Patch 32226239: GI RELEASE UPDATE 19.10.0.0.0

Happy Installing...!!!

Wednesday, February 3, 2021

Oracle Data Guard Broker set up, ORA-12514, ORA-16713, Insufficient SRLs, etc.,

 We have seen how to set up or create an Oracle physical standby database in this post. As a continuation, we will look into how to set up a Data Guard Broker to manage the primary and standby databases. Data Guard Broker provides many advantages such as 

  • enabling configure and manage multiple databases from single location and automatically unifies all DBs in broker configuration, 
  • automatically setting up redo transport services and log apply services, 
  • simplifying switchover and failover, integrating role changes with Oracle clusterware, etc., 
All of the details and advantages can be found in the Oracle Data Guard Broker concepts guide here.

In this post we will look into configuring the Oracle Data Guard Broker and general issues we will encounter during the set up with their work arounds/ fixes. 

Step 1: Preserve current pfile or spfile

As we will be altering/adding a few parameters, it is always a best practice to backup the original pfile/spfile before changing the contents. 

SQL> create pfile='/oracle/ABC/19.0.0/dbs/init_ABC.ora_b4DGBroker' from spfile;

File created.
SQL> 

Step 2
: Starting Data Guard Broker DMON process
  • During physical standby set up we would have set log_archive_dest_2 parameter. We need to clear this parameter where Broker will automatically takes care of this parameter. 
  • If you do not reset the parameter, you will encounter the below error when performing the Create configuration command. 
            Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added
  • We can alter the location of the broker configuration files as needed using dg_broker_config_file1 and dg_broker_config_file2 parameters
  • dg_broker_start should be set to true to let the broker DMON process start automatically when the instance starts up
SQL> ALTER SYSTEM SET dg_broker_config_file1 = '+DATA/ABC1/broker1.dat' scope=both;

System altered.

SQL> ALTER SYSTEM SET dg_broker_config_file2 = '+RECO/ABC1/broker2.dat' scope=both;

System altered.

SQL> alter system reset log_Archive_dest_2 scope=both;

System altered.

SQL> alter system set dg_broker_start=true scope=both;

System altered.

SQL> 

SQL> ALTER SYSTEM SET dg_broker_config_file1 = '+DATA/ABC_STANDBY/broker1.dat' scope=both;

System altered.

SQL> ALTER SYSTEM SET dg_broker_config_file2 = '+RECO/ABC_STANDBY/broker2.dat' scope=both;

System altered.

SQL> alter system reset log_Archive_dest_2 scope=both;

System altered.

SQL> alter system set dg_broker_start=true scope=both;

System altered.

SQL> 
Step 3: Create configuration

Connect to dgmgrl and create configuration as shown below.
-sh-4.2$ dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Jan 21 23:40:44 2021
Version 19.8.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected to "ABC1"
Connected as SYSDBA.

DGMGRL> CREATE CONFIGURATION 'ABC_DGD' AS PRIMARY DATABASE IS 'ABC1' CONNECT IDENTIFIER IS ABC1;
Configuration "ABC_DGD" created with primary database "ABC1"
DGMGRL> show configuration;

Configuration - ABC_DGD

  Protection Mode: MaxPerformance
  Members:
  ABC1 - Primary database

Fast-Start Failover:  Disabled

Configuration Status:
DISABLED

DGMGRL> 
Step 4: Add Standby database
DGMGRL> add database ABC_STANDBY as connect identifier is ABC_STANDBY;
Database "ABC_standby" added
DGMGRL> show configuration;

Configuration - ABC_DGD

  Protection Mode: MaxPerformance
  Members:
  ABC1        - Primary database
    ABC_standby - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
DISABLED

DGMGRL> 
Now you can see the standby database is added but the status is showing as DISABLED

Step 5: Enable configuration
DGMGRL> enable configuration;
Enabled.

DGMGRL> show configuration;

Configuration - ABC_DGD

  Protection Mode: MaxPerformance
  Members:
  ABC1        - Primary database
    ABC_standby - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 8 seconds ago)

DGMGRL> show database ABC1;

Database - ABC1

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    ABC

Database Status:
SUCCESS

DGMGRL> show database ABC_STANDBY;

Database - ABC_standby

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 0 seconds ago)
  Apply Lag:          1 second (computed 0 seconds ago)
  Average Apply Rate: 66.78 MByte/s
  Real Time Query:    OFF
  Instance(s):
    ABC

Database Status:
SUCCESS

DGMGRL> 
We can see the Configuration status is now showing as SUCCESS. 
Line 18 and 30 provides the properties of both Primary and Standby database where the status of both the database is SUCCESS

Step 6: Validate the databases

We can now validate the configuration in which the actual connectivity testing along with a comprehensive set of database checks are being performed. We need to make sure validate database works without any issue for the conversion to take place if intended to. 
DGMGRL> validate database verbose ABC1;

  Database Role:    Primary database

  Ready for Switchover:  Yes

  Flashback Database Status:
    ABC1:  Off

  Capacity Information:
    Database  Instances        Threads
    ABC1      1                1

  Managed by Clusterware:
    ABC1:  NO
    Validating static connect identifier for the primary database ABC1...
Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hic023124.dc.honeywell.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ABC1_DGMGRL)(INSTANCE_NAME=ABC)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Failed.
    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

  Temporary Tablespace File Information:
    ABC1 TEMP Files:  31

  Data file Online Move in Progress:
    ABC1:  No

  Transport-Related Information:
    Transport On:  Yes

  Log Files Cleared:
    ABC1 Standby Redo Log Files:  Cleared

DGMGRL> 
We can see that the validate database is throwing connection error w.r.to connect identifier even though we have properly set up tnsnames and is working fine. The reason being broker adds the static entry in broker configuration as SID_DGMGRL where as the GLOBAL_NAME in listener will be SID which causes the mismatch. Refer Oracle support note Doc ID 1582927.1 for details.
The fix is to set the StaticConnectIdentifier configuration property properly as below 
DGMGRL> edit database ABC1 set property StaticConnectIdentifier='ABC1';
Property "staticconnectidentifier" updated

DGMGRL> edit database ABC_STANDBY set property StaticConnectIdentifier='ABC_STANDBY';                                                                     
Property "staticconnectidentifier" updated

DGMGRL> 
Next we will validate the standby database similar to the above we did for primary database. 
DGMGRL> validate database verbose ABC_STANDBY;
Error: ORA-16713: The Oracle Data Guard broker command timed out.

DGMGRL> 
Now we have a different issue when validating the standby database. Referring Support note Doc ID 1322877.1 and Doc ID 2300040.1 we can extend the Operationtimeout parameter of the broker configuration. The ADR can also be cleaned up prior to running validate database if the database is too huge.
DGMGRL> show configuration OperationTimeout;
  OperationTimeout = '30'
DGMGRL> validate database verbose ABC_STANDBY;
Error: ORA-16713: The Oracle Data Guard broker command timed out.
DGMGRL> EDIT CONFIGURATION SET PROPERTY OperationTimeout=600;
Property "operationtimeout" updated
DGMGRL> validate database verbose ABC_STANDBY;

  Database Role:     Physical standby database
  Primary Database:  ABC1

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Flashback Database Status:
    ABC1       :  Off
    ABC_standby:  Off

  Capacity Information:
    Database     Instances        Threads
    ABC1         1                1
    ABC_standby  1                1

  Managed by Clusterware:
    ABC1       :  NO
    ABC_standby:  NO
    Validating static connect identifier for the primary database ABC1...
    The static connect identifier allows for a connection to database "ABC1".

  Temporary Tablespace File Information:
    ABC1 TEMP Files:         31
    ABC_standby TEMP Files:  30

  Data file Online Move in Progress:
    ABC1:         No
    ABC_standby:  No

  Standby Apply-Related Information:
    Apply State:      Running
    Apply Lag:        0 seconds (computed 0 seconds ago)
    Apply Delay:      0 minutes

  Transport-Related Information:
    Transport On:  Yes
    Gap Status:    No Gap
    Transport Lag:  0 seconds (computed 0 seconds ago)
    Transport Status:  Success

  Log Files Cleared:
    ABC1 Standby Redo Log Files:         Cleared
    ABC_standby Online Redo Log Files:   Cleared
    ABC_standby Standby Redo Log Files:  Available

  Current Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (ABC1)                  (ABC_standby)
    1         8                       9                       Sufficient SRLs

  Future Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (ABC_standby)           (ABC1)
    1         8                       4                       Insufficient SRLs

  Current Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (ABC1)                    (ABC_standby)
    1          3136 MBytes               3136 MBytes

  Future Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (ABC_standby)             (ABC1)
    1          3136 MBytes               3136 MBytes

  Apply-Related Property Settings:
    Property                        ABC1 Value               ABC_standby Value
    DelayMins                       0                        0
    ApplyParallel                   AUTO                     AUTO
    ApplyInstances                  0                        0

  Transport-Related Property Settings:
    Property                        ABC1 Value               ABC_standby Value
    LogShipping                     ON                       ON
    LogXptMode                      ASYNC                    ASYNC
    Dependency                      <empty>                  <empty>
    DelayMins                       0                        0
    Binding                         optional                 optional
    MaxFailure                      0                        0
    ReopenSecs                      300                      300
    NetTimeout                      30                       30
    RedoCompression                 DISABLE                  DISABLE

DGMGRL> 
The command now executed successfully without any issues after configuration change. 
We can see in line 62, broker is reporting insufficient Standby Redo Logs. This is because when the SRLs were added the thread number is not specified in the command. We can drop and recreate the SRLs for which the thread number is not correct to overcome this issue. Refer support note Doc ID 1956103.1 for more details
SQL> select thread#,group#,bytes,status from v$standby_log;

   THREAD#     GROUP#      BYTES STATUS
---------- ---------- ---------- ----------
         1         11 4.1943E+10 UNASSIGNED
         1         12 4.1943E+10 UNASSIGNED
         1         13 4.1943E+10 UNASSIGNED
         1         14 4.1943E+10 UNASSIGNED
         0         15 4.1943E+10 UNASSIGNED
         0         16 4.1943E+10 UNASSIGNED
         0         17 4.1943E+10 UNASSIGNED
         0         18 4.1943E+10 UNASSIGNED
         0         19 4.1943E+10 UNASSIGNED

9 rows selected.

SQL> alter database drop logfile group 15;

Database altered.

SQL> alter database add standby logfile thread 1 group 15 size 40000m;

Database altered.

/* Drop and recreate other groups as well
--alter database drop logfile group 16;
--alter database drop logfile group 17;
--alter database drop logfile group 18;
--alter database drop logfile group 19;

--alter database add standby logfile thread 1 group 16 size 40000m;
--alter database add standby logfile thread 1 group 17 size 40000m;
--alter database add standby logfile thread 1 group 18 size 40000m;
--alter database add standby logfile thread 1 group 19 size 40000m;
*/

SQL> select thread#,group#,bytes/1024/1024, status from v$standby_log;

   THREAD#     GROUP# BYTES/1024/1024 STATUS
---------- ---------- --------------- ----------
         1         11           40000 UNASSIGNED
         1         12           40000 UNASSIGNED
         1         13           40000 UNASSIGNED
         1         14           40000 UNASSIGNED
         1         15           40000 UNASSIGNED
         1         16           40000 UNASSIGNED
         1         17           40000 UNASSIGNED
         1         18           40000 UNASSIGNED
         1         19           40000 UNASSIGNED

9 rows selected.

SQL> 
You can now see that all the SRLs are with proper thread number assigned. Let's check the validate database command for the standby database again. 
DGMGRL> validate database verbose ABC_STANDBY;

  Database Role:     Physical standby database
  Primary Database:  ABC1

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Flashback Database Status:
    ABC1       :  Off
    ABC_standby:  Off

  Capacity Information:
    Database     Instances        Threads
    ABC1         1                1
    ABC_standby  1                1

  Managed by Clusterware:
    ABC1       :  NO
    ABC_standby:  YES
    Validating static connect identifier for the primary database ABC1...
    The static connect identifier allows for a connection to database "ABC1".

  Temporary Tablespace File Information:
    ABC1 TEMP Files:         31
    ABC_standby TEMP Files:  30

  Data file Online Move in Progress:
    ABC1:         No
    ABC_standby:  No

  Standby Apply-Related Information:
    Apply State:      Running
    Apply Lag:        1 second (computed 0 seconds ago)
    Apply Delay:      0 minutes

  Transport-Related Information:
    Transport On:  Yes
    Gap Status:    No Gap
    Transport Lag:  0 seconds (computed 0 seconds ago)
    Transport Status:  Success

  Log Files Cleared:
    ABC1 Standby Redo Log Files:         Cleared
    ABC_standby Online Redo Log Files:   Cleared
    ABC_standby Standby Redo Log Files:  Available

  Current Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (ABC1)                  (ABC_standby)
    1         8                       9                       Sufficient SRLs

  Future Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (ABC_standby)           (ABC1)
    1         8                       9                       Sufficient SRLs

  Current Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (ABC1)                    (ABC_standby)
    1          3136 MBytes               3136 MBytes

  Future Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (ABC_standby)             (ABC1)
    1          3136 MBytes               3136 MBytes

  Apply-Related Property Settings:
    Property                        ABC1 Value               ABC_standby Value
    DelayMins                       0                        0
    ApplyParallel                   AUTO                     AUTO
    ApplyInstances                  0                        0

  Transport-Related Property Settings:
    Property                        ABC1 Value               ABC_standby Value
    LogShipping                     ON                       ON
    LogXptMode                      ASYNC                    ASYNC
    Dependency                      <empty>                  <empty>
    DelayMins                       0                        0
    Binding                         optional                 optional
    MaxFailure                      0                        0
    ReopenSecs                      300                      300
    NetTimeout                      30                       30
    RedoCompression                 DISABLE                  DISABLE

DGMGRL> exit
-sh-4.2$
Everything is set up properly now. We are ready to rock and roll standby management using Oracle Data Guard Broker and the system is ready for switchover and failover operations if needed. 

References

Happy Brokering!!!