Sunday, June 13, 2021

Oracle DBSAT - It's all about security!

 Securing database is inevitable. Companies will strongly focus on securing not only the database but the entire network and everything that's put inside it such as infrastructure, servers, application, web servers, client connections, etc., As we keep on performing tasks related to security, how well we can be confident that we have secured the database to the fullest?

That's where this wonderful tool DBSAT (Database Security Assessment Tool) from Oracle comes to the rescue. This tool is free of cost and can be downloaded from Oracle support site by visiting Doc ID 2138254.1

Pic courtesy: www.oracle.com

In this post, we will see how to set up DBSAT and how to invoke the tool to collect and report security status and to identify sensitive data and improvement opportunities w.r.to security. 

DBSAT consists of 3 components

Collector - Runs queries against DB and OS commands on system to be assessed and stores details in JSON format.

Reporter - Analyzes the collected data and generates the Oracle Database Security Assessment Report in HTML, Excel, JSON, and Text formats

Discoverer - Runs SQL queries and collects data from the system to be assessed, based on the settings specified in the configuration files. Collected data is then used to generate the Oracle Database Sensitive Data Assessment Report in HTML and CSV formats

Here is a small demo I have performed on my lab systems..

Prepare phase: 

Step 1: Download dbsat from Oracle support note Doc ID 2138254.1 and transfer to server where you want to initiate dbsat from.. (central server)

Step 2: Install DBSAT tool (unzip the downloaded file)

-sh-4.2$ mkdir /oracle/media/dbsat
-sh-4.2$ cd /oracle/media/dbsat
-sh-4.2$ ls -lrt
total 4580
-rw-r--r--. 1 oracle oinstall 4687346 Jun  3 06:40 dbsat.zip
-sh-4.2$ unzip dbsat.zip
Archive:  dbsat.zip
  inflating: dbsat
  inflating: dbsat.bat
  inflating: sat_collector.sql
  inflating: sat_reporter.py
  inflating: sat_analysis.py
  inflating: xlsxwriter/app.py
  inflating: xlsxwriter/chart_area.py
  inflating: xlsxwriter/chart_bar.py
  inflating: xlsxwriter/chart_column.py
  inflating: xlsxwriter/chart_doughnut.py
  inflating: xlsxwriter/chart_line.py
  inflating: xlsxwriter/chart_pie.py
  inflating: xlsxwriter/chart.py
  ...
  ...
  ...
  inflating: Discover/conf/sensitive_de.ini
  inflating: Discover/conf/sensitive_pt.ini
  inflating: Discover/conf/sensitive_it.ini
  inflating: Discover/conf/sensitive_fr.ini
  inflating: Discover/conf/sensitive_nl.ini
  inflating: Discover/conf/sensitive_el.ini
-sh-4.2$ ls -lrt
total 5000
-r-xr-xr-x. 1 oracle oinstall   13465 May  5  2020 dbsat
-rw-rw-r--. 1 oracle oinstall  296035 May  6  2020 sat_reporter.py
-rw-rw-r--. 1 oracle oinstall   61823 May  6  2020 sat_collector.sql
-rw-rw-r--. 1 oracle oinstall   26096 May  6  2020 sat_analysis.py
-r-xr-xr-x. 1 oracle oinstall   13644 May  6  2020 dbsat.bat
-rw-r--r--. 1 oracle oinstall 4687346 Jun  3 06:40 dbsat.zip
drwxr-xr-x. 2 oracle oinstall    4096 Jun  3 06:42 xlsxwriter
drwxr-xr-x. 5 oracle oinstall      40 Jun  3 06:42 Discover
-sh-4.2$ 

Step 3: Make sure your tnsping works and you are able to connect to the database to be assessed without issues

-sh-4.2$ tnsping abc.world

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 03-JUN-2021 06:49:08

Copyright (c) 1997, 2020, Oracle.  All rights reserved.

Used parameter files:
/oracle/grid/193/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = abcdb)(PORT = 1527)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ABC)))
OK (0 msec)
-sh-4.2$

-sh-4.2$ mkdir reports

Notice that I have created a directory by name reports under /oracle/media/dbsat and this directory will be used to store the reports generated. 

Step 4: (Optional) Create a user with minimum privileges if you don't want to use user with higher privileges like sys/system
create user dbsat_user identified by dbsat_user;
--If Database Vault is enabled, connect as DV_ACCTMGR to run this command
grant create session to dbsat_user;
grant select_catalog_role to dbsat_user;
grant select on sys.registry$history to dbsat_user;
grant select on sys.dba_users_with_defpwd to dbsat_user; // 11g and 12c
grant audit_viewer to dbsat_user; // 12c
grant capture_admin to dbsat_user;// 12c covers sys.dba_priv_captures, sys.priv_capture$, sys.capture_run_log$ 
--If Database Vault is enabled, connect as DV_OWNER to run this command 
grant DV_SECANALYST to dbsat_user;

Running the collector: 

Now as the set up is completed, we can start the tool to collect the data for our assessment and reporting later. 

The command format is as below

dbsat collect db_connect_string output_file_name

-sh-4.2$ ./dbsat collect system@abc.world ./reports/dbsat-report-abc

Database Security Assessment Tool version 2.2.1 (May 2020)

This tool is intended to assist you in securing your Oracle database
system. You are solely responsible for your system and the effect and
results of the execution of this tool (including, without limitation,
any damage or data loss). Further, the output generated by this tool may
include potentially sensitive system configuration data and information
that could be used by a skilled attacker to penetrate your system. You
are solely responsible for ensuring that the output of this tool,
including any generated reports, is handled in accordance with your
company's policies.

Connecting to the target Oracle database...


SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 3 06:51:07 2021
Version 19.10.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Enter password: --> Enter password for system user

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0

Setup complete.
SQL queries complete.
OS commands complete.
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
DBSAT Collector completed successfully.

Calling /oracle/ABC/193/bin/zip to encrypt dbsat-report-abc.json...

Enter password: --> Enter encryption password, will be used to decrypt later
Verify password:
  adding: dbsat-report-abc.json (deflated 83%)
zip completed successfully.
-sh-4.2$

The collector has run successfully and the o/p file is store by name /oracle/media/dbsat/reports/dbsat-report-abc.json
This report will be used in the next phase for reporting the security status of the database

Running the Reporter:

Note: Reporter requires Python version is 2.6 or later is installed. Install if it's not available on your server

The command format is as below

dbsat report input-file

Options such as -a (report all DB account), -n (no encryption for o/p), -g (show all grants), -x (exclude a section such as user, privilege, etc from report) can be used with the command but I'll stick with the default for now.

-sh-4.2$ ./dbsat report ./reports/dbsat-report-abc

Database Security Assessment Tool version 2.2.1 (May 2020)

This tool is intended to assist you in securing your Oracle database
system. You are solely responsible for your system and the effect and
results of the execution of this tool (including, without limitation,
any damage or data loss). Further, the output generated by this tool may
include potentially sensitive system configuration data and information
that could be used by a skilled attacker to penetrate your system. You
are solely responsible for ensuring that the output of this tool,
including any generated reports, is handled in accordance with your
company's policies.

Archive:  dbsat-report-abc.zip
[dbsat-report-abc.zip] dbsat-report-abc.json password: --> Enter the encryption password provided while running collector
  inflating: dbsat-report-abc.json
DBSAT Reporter ran successfully.

Calling /usr/bin/zip to encrypt the generated reports...

Enter password: --> Enter password to encrypt the reports 
Verify password:
        zip warning: dbsat-report-abc_report.zip not found or empty
  adding: dbsat-report-abc_report.txt (deflated 77%)
  adding: dbsat-report-abc_report.html (deflated 83%)
  adding: dbsat-report-abc_report.xlsx (deflated 3%)
  adding: dbsat-report-abc_report.json (deflated 81%)
zip completed successfully.
-sh-4.2$
-sh-4.2$ cd reports/
-sh-4.2$ ls -lrt
total 176
-rw-------. 1 oracle oinstall  68997 Jun  3 06:56 dbsat-report-abc.zip
-rw-------. 1 oracle oinstall 109062 Jun  3 07:01 dbsat-report-abc_report.zip
-sh-4.2$

Note that to analyze the report we provide the input file as the file generated during the collect phase. We have to provide the password we used during the collect phase to make use of the file. 
Also we have to provide password to encrypt the report files generated. We can use same password for both the encryption to remember easily. 

-- dbsat-report-abc.zip will contain only the collector json file. 
-- dbsat-report-abc_report.zip contains the report in multiple formats. 

You can check here for a sample report (download the file and open in browser of your choice) that I generated for my database which would give a detailed analysis on the security aspects of the database. 

Discoverer: 

Discoverer can be used to generate the Oracle Database Sensitive Data Assessment Report. 
Uses the config file under dbsat_home/Discover/conf. 

Making use of the config file, discoverer can identify the sensitive type data such as identification info (Name, gender, etc.,), biographical info (address, family data, etc.,), financial info (credit card number, bank data, etc.,), job or academic info, etc., so that we can secure the data using different techniques such as masking, encrypting, etc., 
We can also create an exclusion list file if we want to exclude schemas, tables, or columns to exclude from the scan.

For this demo purpose, I have created oracle example schemas in my test system (different from the one used above for collector and reporter, so the DB version will vary) which contains a lot of sensitive data tables/columns which will be reported in the discoverer report.

Command has the following format
dbsat discover -c config_file output_file

My config file contains the below entries. I have just copied the default sample_dbsat.config and named it as orcl_dbsat.config and edited the below connection entries alone and left all as default
[Database]
TNS_ADMIN = /u01/orcl/122/network/admin
NET_SERVICE_NAME = orcl
WALLET_LOCATION =
DB_HOSTNAME = linux75-2
DB_PORT = 1522
DB_SERVICE_NAME = orcl

[oracle@linux75-2 dbsat]$ ./dbsat discover -c ./Discover/conf/orcl_dbsat.config ./reports/orcl-discover

Database Security Assessment Tool version 2.2.1 (May 2020)

This tool is intended to assist you in securing your Oracle database
system. You are solely responsible for your system and the effect and
results of the execution of this tool (including, without limitation,
any damage or data loss). Further, the output generated by this tool may
include potentially sensitive system configuration data and information
that could be used by a skilled attacker to penetrate your system. You
are solely responsible for ensuring that the output of this tool,
including any generated reports, is handled in accordance with your
company's policies.

Enter username: system
Enter password:
DBSAT Discover ran successfully.
Calling /usr/bin/zip to encrypt the generated reports...

Enter password: --> Enter encryption password here
Verify password:
        zip warning: ./reports/orcl-discover_report.zip not found or empty
  adding: reports/orcl-discover_discover.html (deflated 81%)
  adding: reports/orcl-discover_discover.csv (deflated 75%)
Zip completed successfully.
[oracle@linux75-2 dbsat]$ cd reports
[oracle@linux75-2 reports]$ ls -lrt
total 12
-rw-------. 1 oracle oinstall 10450 Jun 13 00:07 orcl-discover_report.zip
[oracle@linux75-2 reports]$

The report will be created in both html and csv format. Pasting the summary picture of the report below and the detailed sample report can be found in this link (download the file and open in browser of your choice)


With this detailed report handy, we can now work on securing the data by using database techniques such as encryption, data masking, etc., That's the end for today :)

References: 


Happy Securing...!!!

Saturday, May 1, 2021

Broken 18c GI upgrade with ORA-29702 and fixes

 Ok, I was very confident the upgrade would succeed when the upgrade started to run after fixing all the necessary pre requisites and is running without issues. I also took a screen shot of the progress as below after starting the rootupgrade.sh See software version already in 18.0.0.0.0


Then after few minutes of my wait, the upgrade failed on node 1. When the rootupgrade.sh script was started and running during Executing upgrade step 17 of 19: 'UpgradeCluster' step, the below error was encountered. 

CRS-2672: Attempting to start 'ora.asm' on '12r1-rac1'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4150
Additional information: 67028943
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/oradir1/grid/base/diag/crs/12r1-rac1/crs/trace/ohasd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.asm' on '12r1-rac1' failed
CRS-2679: Attempting to clean 'ora.asm' on '12r1-rac1'
CRS-2681: Clean of 'ora.asm' on '12r1-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12r1-rac1'
CRS-2883: Resource 'ora.storage' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2021/04/25 17:38:06 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /oradir1/grid/18/crs/install/crsupgrade.pm line 1462.

The alert log of the +ASM1 of 18c home has the following errors. 

MMNL started with pid=29, OS id=26058
2021-04-25T17:27:01.603535+05:30
Unable to start the instance due to software incompatiblity
2021-04-25T17:27:01.604426+05:30
ORA-15156: cluster in rolling upgrade from version [12.1.0.1.0] to [18.0.0.0.0]

2021-04-25T17:27:01.853249+05:30
Error: Shutdown in progress. Error: 29702.
USER (ospid: 25746): terminating the instance due to ORA error 29702
2021-04-25T17:27:02.925386+05:30
Instance terminated by USER, pid = 25746

Detailed info is available in /oradir1/grid/base/diag/crs/12r1-rac1/crs/trace/ohasd_oraagent_oracle.trc file which has reported the below errors.

...
...
...
...
2021-04-25 17:27:14.156 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection:connectInt 200 connected
2021-04-25 17:27:14.156 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 040 resname:ora.asm startup
2021-04-25 17:27:17.534 :CLSFRAME:2796190720:  TM [MultiThread] is changing desired thread # to 5. Current # is 4
2021-04-25 17:27:17.534 :CLSDYNAM:2539628288: [ora.evmd]{0:0:2} [check] DaemonAgent::check returned 0
2021-04-25 17:27:17.534 :CLSDYNAM:2539628288: [ora.evmd]{0:0:2} [check] Deep check returned 0
2021-04-25 17:27:22.679 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] ORA-29702: error occurred in Cluster Group Service operation

2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 050 errcode 29702
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 160 ORA-29701 or ORA-29702 instance dumping m_instanceType:2 m_lastOCIError:29702
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 170 ORA-29701 or ORA-29702 shutdown abort
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::shutdown mode 4
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 260 disConnect
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 020 connect logmode:8008
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::connectInt 020 server not attached
2021-04-25 17:27:33.726 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection:connectInt 200 connected
2021-04-25 17:27:33.727 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 040 resname:ora.asm startup
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] ORA-29702: error occurred in Cluster Group Service operation

2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 050 errcode 29702
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 160 ORA-29701 or ORA-29702 instance dumping m_instanceType:2 m_lastOCIError:29702
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 170 ORA-29701 or ORA-29702 shutdown abort
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::shutdown mode 4
...
...
...

This is a weird error right. Error says unable to start the instance due to software incompatibility and you go make a search on Oracle support for "ORA 29702 during GI upgrade", it returns n number of issues with references to a few bugs as well. Nothing seems to be fitting the issue that I have encountered. 
One more interesting thing is that, if I set the environment variables to 18c home and start the ASM instance, the instance starts without issue. So I then again shutdown the ASM instance and restarted the rootupgrade.sh script. This time too the same error occurs when Oracle tries to start ASM and the upgrade fails. So, after long search and different attempts, I was not able to fix the issue. I cancelled the upgrade session and decided to look into the issue next day. 

On the next day when I start both the nodes, I see that the ASM instance on the first node is up from 18c home and no services started on the second node. I decided to continue the upgrade from where I left. Since we already completed the software set up and our error occurred during the execution of rootupgrade.sh script, I started from the same step after shutting down the entire cluster running on node 1. 
This time the rootupgrade.sh completed on node 1. Shut down and startup of the entire server did the trick it seems. 

So I continued with the upgrade by running the rootupgrade.sh on node 2 which also succeeded without any issues. Finally ran the below command to complete the upgrade. 

$GRID_HOME/gridSetup.sh -executeConfigTools -all -silent -responseFile responsefile.rsp

Well, now after everything has been done, the command ended with the below error. 

You can find the logs of this session at:
/u01/oraInventory/logs/GridSetupActions2021-04-29_01-25-13AM

Configuration failed.
[WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped.
   ACTION: Refer to the logs or contact Oracle Support Services.

$ 

We can also see the below response when querying the active version

[oracle@12r1-rac1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
[oracle@12r1-rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [UPGRADE FINAL]. The cluster active patch level is [70732493].
[oracle@12r1-rac1 ~]$

This is a new change in GI starting 18c where cluvfy performs a post upgrade check and reports the status and doesn't change the status from [UPGRADE FINAL] to [NORMAL] if any of the post upgrade check fails. 

In my case, opening the detailed log /u01/oraInventory/logs/GridSetupActions2021-04-29_01-25-13AM/gridSetupActions2021-04-29_01-25-13AM.log shows the below errors (trimmed output)

...
...
...
INFO:  [Apr 29, 2021 1:42:07 AM] Skipping line: Verifying Single Client Access Name (SCAN) ...PASSED
INFO:  [Apr 29, 2021 1:42:11 AM] Verifying OLR Integrity ...PASSED
INFO:  [Apr 29, 2021 1:42:11 AM] Skipping line: Verifying OLR Integrity ...PASSED
INFO:  [Apr 29, 2021 1:42:17 AM] Verifying Voting Disk ...FAILED (PRVF-5431, PRVF-5449, PRVG-2032)
INFO:  [Apr 29, 2021 1:42:17 AM] Skipping line: Verifying Voting Disk ...FAILED (PRVF-5431, PRVF-5449, PRVG-2032)
INFO:  [Apr 29, 2021 1:42:17 AM] Verifying ASM Integrity ...
INFO:  [Apr 29, 2021 1:42:17 AM] Skipping line: Verifying ASM Integrity ...
INFO:  [Apr 29, 2021 1:42:17 AM]   Verifying Node Connectivity ...
...
...
...
INFO:  [Apr 29, 2021 1:42:22 AM] Skipping line: Verifying Device Checks for ASM ...
INFO:  [Apr 29, 2021 1:42:23 AM]   Verifying Access Control List check ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:23 AM] Skipping line:   Verifying Access Control List check ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:28 AM]   Verifying I/O scheduler ...PASSED
INFO:  [Apr 29, 2021 1:42:28 AM] Skipping line:   Verifying I/O scheduler ...PASSED
INFO:  [Apr 29, 2021 1:42:28 AM] Verifying Device Checks for ASM ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:28 AM] Skipping line: Verifying Device Checks for ASM ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:30 AM] Verifying ASM disk group free space ...PASSED
INFO:  [Apr 29, 2021 1:42:30 AM] Skipping line: Verifying ASM disk group free space ...PASSED
INFO:  [Apr 29, 2021 1:42:30 AM] Verifying User Not In Group "root": oracle ...PASSED
...
...
...

According to support note Doc ID 2583141.1, fix the critical issues that cluvfy post upgrade check reports and rerun cluvfy with -collect cluster to update the status to [NORMAL]. 

[oracle@12r1-rac1 ~]$ $ORACLE_HOME/bin/cluvfy stage -post crsinst -gi_upgrade -n all

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.56.0" ...PASSED
Verifying Node Connectivity ...PASSED
...
...
...
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...FAILED (PRVF-9992)
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...FAILED (PRVF-9992)
...
...
...
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following nodes:
        12r1-rac2,12r1-rac1


Failures were encountered during execution of CVU verification request "stage -post crsinst".

Verifying Voting Disk ...FAILED
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed

12r1-rac2: PRVF-5449 : Check of Voting Disk location "/dev/oracleasm/disk1"
           failed on the following nodes:
12r1-rac2: PRVG-2032 : Group of file "/dev/oracleasm/disk1" did not match the
           expected value on node "12r1-rac2". [Expected = "oinstall" ; Found =
           "dba"]

12r1-rac1: PRVF-5449 : Check of Voting Disk location "/dev/oracleasm/disk1"
           failed on the following nodes:
12r1-rac1: PRVG-2032 : Group of file "/dev/oracleasm/disk1" did not match the
           expected value on node "12r1-rac1". [Expected = "oinstall" ; Found =
           "dba"]

Verifying Device Checks for ASM ...FAILED
  Verifying Access Control List check ...FAILED
  12r1-rac2: PRVF-9992 : Group of device "/dev/oracleasm/disk5" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
             nodes: [12r1-rac2, 12r1-rac1]
  12r1-rac2: PRVF-9992 : Group of device "/dev/oracleasm/disk4" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
...
...
...
  12r1-rac1: PRVF-9992 : Group of device "/dev/oracleasm/disk8" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
             nodes: [12r1-rac2, 12r1-rac1]


CVU operation performed:      stage -post crsinst
Date:                         May 1, 2021 7:37:41 PM
CVU home:                     /oradir1/grid/18/
User:                         oracle
[oracle@12r1-rac1 ~]$

We can see most of the errors are due to group expectation on the ASM disks. I just changed the disk ownership to oracle:oinstall instead of dba by editing the udev rules and reloading and reran the cluvfy for post upgrade which succeeded without issues. 
Next step is to run cluvfy with -collect cluster to update the status of the GI upgrade (trimmed output)

[oracle@12r1-rac1 ~]$ $ORACLE_HOME/bin/cluvfy stage -post crsinst -collect cluster -gi_upgrade -n all

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.56.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
...
...
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED

Collecting OS mandatory requirements baseline

Collecting Physical Memory ...collected
Collecting Available Physical Memory ...collected
...
...
...
Collecting Package: ksh ...collected
Collecting Package: make-3.81 ...collected
...
...
...
Collecting HugePages Existence ...collected
Collecting Hardware Clock synchronization at shutdown ...collected
Collecting availability of port 8888 ...collected
Collecting Ethernet Jumbo Frames ...collected
...
...
...
Collecting CSS reboottime parameter ...collected
Collecting CSS disktimeout parameter ...collected


Verifying ASM mandatory requirements

Collecting ASM Initialization Parameters ...collected
Collecting ASM Configuration Collection ...collected
Collecting ASM Disk Group Configuration Collection ...collected


Verifying ASM best practices

Collecting ASM_POWER_LIMIT ...collected
...
...
...
Collecting ASM disk rebalance operations in WAIT status ...collected

Baseline collected.
Collection report for this execution is saved in file "/oradir1/grid/base/crsdata/@global/cvu/baseline/install/grid_install_18.0.0.0.0.zip".

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         May 1, 2021 8:22:45 PM
CVU home:                     /oradir1/grid/18/
User:                         oracle
[oracle@12r1-rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [70732493].
[oracle@12r1-rac1 ~]$

We can see now the cluster upgrade state is [NORMAL] meaning the GI upgrade is a success!

Foot Note: 
This blog post is the one which is closest match to the error which we encountered where the author says regarding Bug 21484367 but I doubt this might have matched as I have not configured so many services as this system being my home lab. Also ASM instance came up without issues after starting the node the next day (the author says after he killed ohasd and oraagent.bin process the GI and evmd came up). I didn't get a chance to try this anyhow. 

References: 
  • How to resolve the cluster upgrade state of [UPGRADE FINAL] after successfully upgrading Grid Infrastructure (GI) to 18c or higher (Doc ID 2583141.1)
  • How to Install / Upgrade/ Clone 12.2 Grid Infrastructure in Silent Mode Using gridSetup.sh (Doc ID 2327772.1)
  • evmd not starting in oracle restart (pfierens.blogspot.com)
  • Various Oracle support notes popping up with search term "ORA 29702 during GI upgrade"

Happy upgrading...!!!