Enterprise Manager Cloud Control - Blackout - manual update in OMS

I have created the blackout for one of the servers with duration of 2 hours. Even after a month, still the server pops up in the Blackouts window of the EMCC. It is clearly seen the status is in “Stop Failed” 

There is no way we can delete the blackout via GUI as the option is grayed out. We have an option of stop but the result of stop is as below. 

We can even try to stop or delete the black out via command lines but still we will get the below errors. 

-sh-4.2$ 
-sh-4.2$ emcli get_blackout_details -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Status                       Status ID  Run Jobs  Next Start  Duration  Reason                   Frequency  Repeat  Days  Months  Start Time           End Time             TZ Region  TZ Offset  Type
Full Blackout - Stop Failed  6          no        none        02:00     Added via emctl utility  once       none    none  none    2020-08-16 10:05:06  2020-08-16 12:05:06  Etc/GMT    +00:00     Blackout
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli stop_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Timed out while waiting for blackout to stop. Please check the blackout status from Enterprise Manager Cloud Control
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Cannot delete a blackout that has not ended or was not stopped
-sh-4.2$

This is because, after the blackout has been started, we have stopped the agent and the server was put for decommission. Hence there is no agent to which the OMS can talk to perform stop or even delete the blackout created when the server/agent was alive.

Though this is not going to harm the EMCC operations in any way, the blackout will be listed as ACTIVE though we know that’s not the case. So better to end the blackout and cleanup the mess.

So how do we get out of this situation?

There are a few notes in Oracle support as below and a few more

EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)

These notes doesn’t apply to this scenario as we don’t have the target system for OMS to connect with.

We need to update the status manually for the blackouts which are hanging in the OMS. The work around provided in the support note Bug 30708506 : Remove historical blackouts in a scheduled status can be used to alter the status of the blackout to put it either to stopped (10) state or to ended (11) state after which the blackout can be deleted if needed.

The below operation has to be done on the OMS repository database. 

-sh-4.2$ 
-sh-4.2$ sqlplus sysman/*****

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 2 09:33:45 2020
Version 19.8.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Last Successful login time: Mon Nov 02 2020 09:32:39 +00:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL>
SQL> SELECT UNIQUE target_name
        ,blackout_name
        ,STATUS
        ,start_time
        ,end_time
        ,target_type
FROM (
        SELECT a.blackout_name
                ,a.start_time
                ,a.end_time
                ,a.STATUS
                ,a.target_name
                ,a.target_type
        FROM sysman.MGMT$BLACKOUT_HISTORY a
        )
WHERE STATUS = 'Stop Failed'
ORDER BY start_time;  2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17

no rows selected

SQL> select blackout_guid from em_blackouts_ng where blackout_name='CMQ_2020-08-16 10:05:06';

BLACKOUT_GUID
--------------------------------
1A0C988C8BC4932A6D14F2B36E63D472

SQL> UPDATE em_blackout_target_state SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> UPDATE em_blackouts_ng SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> commit;

Commit complete.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
-sh-4.2$ 

You can now see the status is updated as “Ended” in EMCC as well.

An attempt to delete the blackout now gets completed successfully

-sh-4.2$ 
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://*****:3872/emd/main/"
Blackout "CMQ_2020-08-16 10:05:06" deleted successfully
-sh-4.2$

The clean up is now done and we can peacefully sleep :)

References: 

EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)
Bug 30708506 : Remove historical blackouts in a scheduled status

https://docs.oracle.com/cd/E63000_01/EMVWS/blackoutviews.htm#EMVWS32043

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emcli/em-cli-verbs.html

Happy Work arounding!!! 

Create a standby database using RMAN active duplicate - Oracle 19c

My previous posts are about how to clone Grid home and RDBMS home. 

In this post we will look at how to create a standby database using RMAN duplicate command. 

I've already written about the standby database creation in this post a few years back using Oracle 9i database. The major difference to the old post is that we are now going to perform standby build using active database. Let's look into how to perform the same. The old post is still valid for building standby using the same steps explained even for 19c database.

We still have to satisfy all the prerequisites before building a standby database. Prerequisites basically comprises of enabling archive log mode and force logging, creating and copying the password file to auxiliary instance, etc. You can go through and perform all the activities defined in this post to complete all prerequisites. We can skip performing the database backup for this activity as we are now going to use active database to perform the standby build. 

This is going to be very simple compared to the post which I created for Oracle 9i. 

Few checks: 

# Copy the password file from primary and store in standby server and rename to match standby naming

From Primary: 

scp $ORACLE_HOME/dbs/orapwABC standbyserver:/oracle/ABC/19.0.0/dbs/

# Prepare Primary for standby setup

# Add below parameters in primary

alter system set LOG_ARCHIVE_CONFIG='DG_CONFIG=(ABC,ABC_STANDBY)' scope=both;

alter system set LOG_ARCHIVE_DEST_1='LOCATION=+ARCH/ABC/oraarch VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=ABC' scope=both;

alter system set LOG_ARCHIVE_DEST_2='SERVICE=ABC_STANDBY LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ABC_STANDBY' scope=both;

alter system set LOG_ARCHIVE_DEST_STATE_1=ENABLE scope=both;

alter system set LOG_ARCHIVE_DEST_STATE_2=ENABLE scope=both;

alter system set FAL_SERVER=ABC_STANDBY scope=both;

alter system set FAL_CLIENT=ABC scope=both;

# Try tnsping in both target server and auxiliary server and both should work

From Primary: tnsping abc_standby
From standby: tnsping abc

### This is from Primary server
-sh-4.2$ tnsping abc_standby

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 19-AUG-2020 01:25:46

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:
/usr/sap/ABC/SYS/profile/oracle/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = xxxxx) (PORT = 1527))) (CONNECT_DATA = (SID = ABC) (GLOBAL_NAME = ABC.WORLD)))
OK (70 msec)
-sh-4.2$ 

### This is from Standby server
-sh-4.2$ tnsping abc

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 19-AUG-2020 01:26:46

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:
/oracle/ABC/19.0.0/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = abcdb) (PORT = 1527))) (CONNECT_DATA = (SID = ABC) (GLOBAL_NAME = ABC.WORLD)))
OK (0 msec)
-sh-4.2$

Assuming we have already completed all the prerequisites, I'm going to start the auxiliary instance with just the below parameters. 

# Create a simple pfile (initABC.ora) under $ORACLE_HOME/dbs with the below entries 

*.db_name=ABC

*.db_unique_name=ABC_STANDBY

*.db_block_size=16384 

-- For db_block_size choose which is optimal for your environment. General and most common is 8kb

# Create all necessary directories for trace files etc. This can be similar set up to your primary database server as names of both the database will be same. 

# Start standby database in nomount 

-sh-4.2$ . oraenv
ORACLE_SID = [ABC] ?
The Oracle base has been changed from /oracle to /oracle/ABC
-sh-4.2$ sqlplus / as sysdba

SQL> startup nomount pfile=/oracle/ABC/19.0.0/dbs/initABC.ora

# Check connectivity from both ends

From Primary: sqlplus sys/<passwd<@ABC AS SYSDBA

From Standby: sqlplus sys/<passwd>@ABC_STANDBY AS SYSDBA

Create standby script

$ more create_abcstby.ksh
#!/bin/ksh
dt=`date '+%Y%m%d'`
mylog=create_abcstby_${dt}.log
touch $mylog
rman target sys/*****@ABC auxiliary sys/*****@ABC_STANDBY cmdfile create_abcstby.rcv msglog $mylog
$

Contents of create_abcstby.ksh is as follows

$ more create_abcstby.rcv
run {
ALLOCATE CHANNEL t1 DEVICE TYPE disk;
ALLOCATE CHANNEL t2 DEVICE TYPE disk;
ALLOCATE CHANNEL t3 DEVICE TYPE disk;
ALLOCATE CHANNEL t4 DEVICE TYPE disk;

ALLOCATE AUXILIARY CHANNEL a1 DEVICE TYPE disk;
ALLOCATE AUXILIARY CHANNEL a2 DEVICE TYPE disk;
ALLOCATE AUXILIARY CHANNEL a3 DEVICE TYPE disk;
ALLOCATE AUXILIARY CHANNEL a4 DEVICE TYPE disk;

duplicate target database
        for standby
        from active database
        dorecover
nofilenamecheck
spfile
set db_name='ABC'
set db_unique_name='ABC_STANDBY'
set audit_file_dest='/oracle/ABC/saptrace/audit'
set diagnostic_dest='/oracle/ABC/saptrace'
set control_files='+DATA','+LOG1','+LOG2'
set db_create_file_dest='+DATA'
set db_create_online_log_dest_1='+LOG1'
set db_create_online_log_dest_2='+LOG2'
set db_recovery_file_dest='+RECO'
set db_recovery_file_dest_size='10240G'
set log_archive_max_processes='8'
set fal_client='ABC_STANDBY'
set fal_server='ABC'
set standby_file_management='AUTO'
set log_archive_config='dg_config=(ABC,ABC_STANDBY)'
set log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=ABC_STANDBY'
set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=xxxxxxx)(PORT=1527))'
set log_archive_dest_2='service=ABC db_unique_name=ABC valid_for=(online_logfile, primary_role)'
set sga_max_size='1280G'
;
}

Script explained: 

1. We started the auxiliary instance with just the 3 parameters required
2. Once we start the script RMAN will be connecting to both Target (Primary) and Auxiliary (Standby) database to perform the duplicate.
3. RMAN will use the contents of the spfile used in the Primary database to start the auxiliary database. 
4. The set commands specified in the RMAN duplicate script will introduce new parameters to the auxiliary instance if not present in the primary instance and override any parameters with the new values if the parameters already exists in the primary instance. 
5. For example, db_unique_name will be ABC in the primary instance and now will be override by ABC_STANDBY in the auxiliary instance. Like wise if we don't have log_archive_max_processes process defined in primary instance, this process will be introduced to the auxiliary (standby) instance. 

Note that I have allocated equal number of channels for both target channels and auxiliary channels. This is an RMAN active duplicate feature enhancement introduced in Oracle version 12c and above to off load processing of duplicate operation to the auxiliary site. 

When the number of auxiliary channels are allocated equal to the number of target channels, the backup request is initiated by auxiliary channels and processing load is done in the auxiliary instance. This is called as pull-based method of active duplication which differs from the push-based method of duplication which uses image copies of the target DB to be copied to auxiliary using target channels and is done by target instance. To know in detail about push bases and pull based method, please check here

We can start the activity by calling the script in nohup mode as we are not sure how long the script will run and we don't any disconnects to disrupt our activity. 

# Run the below in auxiliary (standby) server

$ nohup create_bwpstby.ksh &

Once the script has completed execution the database will be in mount state and recovered until the logs are available. 

The last step is to enable MRP process which will then keep the standby database in sync with primary database by applying the logs as and when it receives the logs.

 -- Start MRP process 
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

Media recovery complete.
SQL> 

Starting 12.1.0.2, the above command will start real time apply if we have Standby Redo Logs (SRL) available in the standby database. Since we already created the SRLs as part of our prerequisites, we should now have real time changes applied on the standby database. This also deprecates use of "USING CURRENT LOGFILE" clause from the real time apply command and throws a warning message in alert log file if used. 

What if we don't have SRLs created? No worries, dataguard will still work but will apply archived logs as and when available in the standby site as opposed to the online redo changes apply. This will have a delay in standby data sync with respect to primary and we can see the below warning in the alert log file as well. 

WARNING: There are no standby redo logs.

Standby redo logs should be configured for real time apply. Real time apply will be ignored.

Now once all the set up is completed, we can check the status of the dataguard using the below command. 

# Verify DG works

select process, sequence#, status from v$managed_standby;

Some tips... 

Since I'm using OMF, we don't have to include log_file_name_convert and db_file_name_convert when duplicating the database

We have an enhanced parameter parameter_value_convert starting Oracle version 11g which can update all string values, not just those containing path names. The values are case-sensitive.

References: 

Duplicating a Database - I have used 12.1 document to show the new features

Oracle Support note: Step by Step Guide on Creating Physical Standby Using RMAN DUPLICATE...FROM ACTIVE DATABASE (Doc ID 1075908.1)

Oracle Support note: RMAN ACTIVE DUPLICATE USING BACKUPSET IN 12C(NEW FEATURE) (Doc ID 1987193.1)

Oracle Support note: Rman Active Duplicate Runs Into RMAN-06217 -- PUSH & PULL method Explanation (Doc ID 2369137.1)

Happy standby building!!! :)

Oracle - Cloning rdbms home

In my previous post I have explained on how to clone a grid home. Let's look at how to clone RDBMS home in this post. This is really simple compared to the cloning of grid home..

Step 1: Create RDBMS Gold image

Set the database home environment and run createGoldImage using runInstaller

$ $ORACLE_HOME/runInstaller -silent -createGoldImage -destinationLocation /oracle/media/Grid_GoldImg/19.7_goldImage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /oracle/media/Grid_GoldImg/19.7_goldImage/db_home_2020-07-27_11-28-33PM.zip

$

Once the software zip is created, it can be transferred to the target server and all the steps below are to be executed on the target server

Step 2: Unzip the Gold image

$ mkdir -p /oracle/ABC/19.0.0
$ cd /oracle/ABC/19.0.0
$ ls -lrt /oracle/media/Grid_GoldImg/19.7_goldImage/
total 9163480
-rwxrwxrwx. 1 oracle oinstall 5290166783 Jul 28 06:36 grid_home_2020-07-27_11-24-58PM.zip
-rwxrwxrwx. 1 oracle oinstall 4093232595 Jul 28 06:39 db_home_2020-07-27_11-28-33PM.zip
$ unzip /oracle/media/Grid_GoldImg/19.7_goldImage/db_home_2020-07-27_11-28-33PM.zip
Archive:  /oracle/media/Grid_GoldImg/19.7_goldImage/db_home_2020-07-27_11-28-33PM.zip
   creating: R/
   creating: R/library/
   creating: R/library/ORE/
  inflating: R/library/ORE/DESCRIPTION
   creating: R/library/ORE/doc/
   creating: R/library/ORE/doc/man/
   creating: R/library/ORE/doc/man/de/
  inflating: R/library/ORE/doc/man/de/OREShowDoc.Rd
   creating: R/library/ORE/doc/man/en/
....
....
finishing deferred symbolic links:
  bin/lbuilder           -> ../nls/lbuilder/lbuilder
  lib/libobk.so          -> /usr/openv/netbackup/bin/libsapora.so64
  lib/libocci.so         -> libocci.so.19.1
  lib/libodm19.so        -> libodmd19.so
  lib/libagtsh.so        -> libagtsh.so.1.0
  lib/libclntsh.so       -> libclntsh.so.19.1
  lib/libjavavm19.a      -> ../javavm/jdk/jdk8/lib/libjavavm19.a
....
....
  javavm/lib/security/README.txt -> ../../../javavm/jdk/jdk8/lib/security/README.txt
  javavm/lib/sunjce_provider.jar -> ../../javavm/jdk/jdk8/lib/sunjce_provider.jar
  javavm/lib/security/java.security -> ../../../javavm/jdk/jdk8/lib/security/java.security
$

Step 3: Install the software

Prepare the response file and install the software. Remember, as mentioned in the previous post usage of clone.pl script is deprecated. Sample response file is here db.rsp

$ ./runInstaller -silent -responseFile /home/oracle/db.rsp
Launching Oracle Database Setup Wizard...

The response file for this session can be found at:
 /oracle/ABC/19.0.0/install/response/db_2020-07-30_06-27-30AM.rsp

You can find the log of this install session at:
 /oracle/oraInventory/logs/InstallActions2020-07-30_06-27-30AM/installActions2020-07-30_06-27-30AM.log

As a root user, execute the following script(s):
        1. /oracle/ABC/19.0.0/root.sh

Execute /oracle/ABC/19.0.0/root.sh on the following nodes:
[XXXXXXX]


Successfully Setup Software.
$ sudo /oracle/ABC/19.0.0/root.sh
Check /oracle/ABC/19.0.0/install/root_XXXXXXX.dc.honeywell.com_2020-07-30_06-39-56-995239503.log for the output of root script
$ cat /oracle/ABC/19.0.0/install/root_XXXXXXX.dc.honeywell.com_2020-07-30_06-39-56-995239503.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/ABC/19.0.0
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA) is available at : /oracle/ABC/19.0.0/bin/tfactl
$

You can verify whether the home has been added to the inventory by reading the inventory.xml file

$ cat /oracle/oraInventory/ContentsXML/inventory.xml





   12.2.0.7.0
   2.1.0.6.0








$

Now we have successfully installed the RDBMS software as well. 

Just a tip... 

We can still use clone.pl script to clone the home using old method but you will get a warning message before proceeding as below

$ /oracle/ABC/19.0.0/perl/bin/perl /oracle/ABC/19.0.0/clone/bin/clone.pl -silent ORACLE_HOME=/oracle/ABC/19.0.0 ORACLE_BASE=/oracle/ABC -defaultHomeName INVENTORY_LOCATION=/oracle/oraInventory


[INFO] [INS-32183] Use of clone.pl is deprecated in this release. Clone operation is equivalent to performing a Software Only installation from the image.
You must use /oracle/ABC/19.0.0/runInstaller script available to perform the Software Only install. For more details on image based installation, refer to help documentation.

Starting Oracle Universal Installer...

You can find the log of this install session at:
 /oracle/oraInventory/logs/cloneActions2020-06-03_02-18-22PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
...
...
...
Finish Setup successful.
The cloning of OraHome1 was successful.
Please check '/oracle/oraInventory/logs/cloneActions2020-06-03_02-18-22PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
        1. /oracle/ABC/19.0.0/root.sh



..................................................   100% Done.
$

It's always better to use the updated scripts instead of using the outdated/deprecated scripts.. 

Happy cloning!!!

Oracle - Cloning grid home

In this post we will look into how to clone an existing Grid home on another server. There are many advantages in doing this as this saves time. We can create a single home by installation process in a server and clone the installation along with all the patches that are applied to the installed server on to multiple servers without the need of going through installation and patching process again. 

You can refer the below link in order to learn more about the cloning process and its advantages. 

Cloning Oracle Clusterware

Note the link refers to Oracle version 11.2 and in this post I'll be taking you through the process with version 19.7. 

clone.pl script referred in the document is deprecated from Oracle 19c and hence we will use root.sh to create the new grid home. 

Environment details:
OS: Red Hat Enterprise Linux Server release 7.7 (Maipo)
Oracle: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.7.0.0.0

Step 1: Create the Oracle Grid Home Gold image

Set the environment variables to +ASM instance and run the below on the source server where Oracle database server software is already installed

In the reference 11.2 document provided above, Oracle suggests that you remove unnecessary files and create an exclude list before copy of the Oracle home. In the recent version, they are smartly done by Oracle itself and the zip file is also create by creategoldimage command. 

$ $ORACLE_HOME/gridSetup.sh -creategoldimage -destinationlocation /oracle/media/Grid_GoldImg/19.7_goldImage -silent
Launching Oracle Grid Infrastructure Setup Wizard...

Successfully Setup Software.
Gold Image location: /oracle/media/Grid_GoldImg/19.7_goldImage/grid_home_2020-07-27_11-24-58PM.zip

Once the software zip is created, it can be transferred to the target server and all the steps below are to be executed on the target server

Step 2: Check all Oracle Prerequisites

Like in a normal Oracle database server installation, you should satisfy all the Oracle installation prerequisites. Refer to Database Installation guide for Linux for full and proper prerequisites. For easy reference please satisfy all the points mentioned in the document here Prereq-linux-19c

Along with the prereq document, the user and group requirements are to be satisfied as well. Simple command for reference is as below. Modify according to your environment

# # CREATE ORACLE ACCOUNT
# useradd -u 600 -g oinstall -s /bin/bash -d /home/oracle -m oracle
#
# # MODIFY ORACLE ACCOUNT
# usermod -a -G dba,oper,asmadmin,asmoper,asmdba oracle

Step 3: Unzip GRID binaries to grid home

Create the grid home directory and unzip the transferred Gold image zip file

$ mkdir -p /oracle/grid/19.0.0
$ cd /oracle/grid/19.0.0
$ unzip /oracle/media/Grid_GoldImg/19.7_goldImage/grid_home_2020-07-27_11-24-58PM.zip
Archive:  /oracle/media/Grid_GoldImg/19.7_goldImage/grid_home_2020-07-27_11-24-58PM.zip
   creating: cv/
   creating: cv/admin/
  inflating: cv/admin/cvunetquery.bin
  inflating: cv/admin/cvusys.sql
  inflating: cv/admin/cvu_config
  inflating: cv/admin/odnsdlite.bin
  inflating: cv/admin/cvunetquery
  inflating: cv/admin/odnsdlite
   creating: cv/cvdata/
   creating: cv/cvdata/101/
  inflating: cv/cvdata/101/crsinst_prereq.xml
  inflating: cv/cvdata/101/dbcfg_prereq.xml
  inflating: cv/cvdata/101/dbinst_prereq.xml
....
....
  inflating: install/.img.bin
finishing deferred symbolic links:
  bin/lbuilder           -> ../nls/lbuilder/lbuilder
  lib/libocci.so         -> libocci.so.19.1
  lib/libodm19.so        -> libodmd19.so
  lib/libagtsh.so        -> libagtsh.so.1.0
....
....
  javavm/lib/security/README.txt -> ../../../javavm/jdk/jdk8/lib/security/README.txt
  javavm/lib/sunjce_provider.jar -> ../../javavm/jdk/jdk8/lib/sunjce_provider.jar
  javavm/lib/security/java.security -> ../../../javavm/jdk/jdk8/lib/security/java.security

Step 4: Install cvuqdisk rpm

cvuqdisk rpm will be bundled along with Oracle software. Install as root user or use sudo

$ cd cv/rpm/
$ ls -lrt
total 12
-rw-r--r--. 1 oracle oinstall 11620 Apr 11 08:19 cvuqdisk-1.0.10-1.rpm
$ sudo rpm -ivh cvuqdisk-1.0.10-1.rpm
Preparing...                          ################################# [100%]
Using default group oinstall to install package
Updating / installing...
   1:cvuqdisk-1.0.10-1                ################################# [100%]
$

Step 5: Run cluster verify (runcluvfy.sh) script 

runcluvfy.sh can be run to find whether all the prerequisites are properly satisfied before we run the grid setup

$ cd
$ /oracle/grid/19.0.0/runcluvfy.sh stage -pre hacfg -verbose > mycluvfy.out
$ tail -f mycluvfy.out
...
...
Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying OS Kernel Parameter: semopm ...FAILED
xxxxxx: PRVG-1205 : OS kernel parameter "semopm" does not have expected
           current value on node "xxxxxx" [Expected = "100" ; Current =
           "32"; Configured = "undefined"].

...
...

The above shows an example of failed prerequisite required for the installation. If all the prerequisites are taken care/fulfilled, this command will not fail for any prerequisites. Otherwise, the command will suggest to alter settings, install required rpms, etc. Those should be satisfied before proceeding further. 

Please check the mycluvfy.out file for failures and fix them.

Once all prerequisites satisfied, the below will be logged meaning a successful prerequisite checks

Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  xxxxxx                                passed
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was successful.

CVU operation performed:      stage -pre hacfg
Date:                         Jul 29, 2020 5:32:56 AM
CVU home:                     /oracle/grid/19.0.0/
User:                         oracle
$ 

Step 6: Run gridSetup.sh

Prepare the response file and run the gridSetup.sh. Previously this used to be clone.pl but now its replaced by gridSetup.sh which runs the already compiled grid home with all the patches included.

Example response file grid.rsp

$ /oracle/grid/19.0.0/gridSetup.sh -silent -responseFile /home/oracle/grid.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-30543] Installer has detected that one or more of the selected disk(s) is of size larger than 2 terabyte(TB). Because of this, the Oracle Database compatibility level (COMPATIBLE.RDBMS attribute) of the diskgroup (DATA) will be set to 12.1. This means that, in order to use the diskgroup for data storage, the COMPATIBLE initialization parameter of the Oracle Database should be greater than or equal to 12.1.
   CAUSE: The following disks are of size larger than 2 terabyte(TB): [/dev/oracleasm/DATA1, /dev/oracleasm/DATA2
[WARNING] [INS-30011] The SYS password entered does not conform to the Oracle recommended standards.
   CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
   ACTION: Provide a password that conforms to the Oracle recommended standards.
[WARNING] [INS-30011] The ASMSNMP password entered does not conform to the Oracle recommended standards.
   CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
   ACTION: Provide a password that conforms to the Oracle recommended standards.
[WARNING] [INS-32047] The location (/oracle/oraInventory) specified for the central inventory is not empty.
   ACTION: It is recommended to provide an empty location for the inventory.
The response file for this session can be found at:
 /oracle/grid/19.0.0/install/response/grid_2020-07-29_10-04-33AM.rsp

You can find the log of this install session at:
 /tmp/GridSetupActions2020-07-29_10-04-33AM/gridSetupActions2020-07-29_10-04-33AM.log

As a root user, execute the following script(s):
        1. /oracle/oraInventory/orainstRoot.sh
        2. /oracle/grid/19.0.0/root.sh

Execute /oracle/grid/19.0.0/root.sh on the following nodes:
[XXXXXX]



Successfully Setup Software.
As install user, execute the following command to complete the configuration.
        /oracle/grid/19.0.0/gridSetup.sh -executeConfigTools -responseFile /home/oracle/grid.rsp [-silent]


Moved the install session logs to:
 /oracle/oraInventory/logs/GridSetupActions2020-07-29_10-04-33AM
$ 

Make sure to provide good password to avoid the above warnings. 

Step 7: Execute the scripts as root and oracle user as directed by oracle above

Highlighted lines are the commands executed on the terminal

$ # Switching to root user
$ sudo su -
# /oracle/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/oraInventory to oinstall.
The execution of the script is complete.
# /oracle/grid/19.0.0/root.sh
Check /oracle/grid/19.0.0/install/root_XXXXXX.dc.honeywell.com_2020-07-29_10-09-01-095734828.log for the output of root script
#
#
# cat /oracle/grid/19.0.0/install/root_XXXXXX.dc.honeywell.com_2020-07-29_10-09-01-095734828.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/grid/19.0.0
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/grid/19.0.0/crs/install/crsconfig_params
The log of current session can be found at:
  /oracle/grid/crsdata/xxxxxx/crsconfig/roothas_2020-07-29_10-09-01AM.log
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node xxxxxx successfully pinned.
2020/07/29 10:09:13 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

xxxxxx     2020/07/29 10:13:26     /oracle/grid/crsdata/xxxxxx/olr/backup_20200729_101326.olr     3030781831
2020/07/29 10:13:27 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
# exit
$ # Running the next script as Oracle user 
$ /oracle/grid/19.0.0/gridSetup.sh -executeConfigTools -responseFile /home/oracle/grid.rsp -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/oracle/oraInventory/logs/GridSetupActions2020-07-29_10-16-06AM

You can find the log of this install session at:
 /oracle/oraInventory/logs/UpdateNodeList2020-07-29_10-16-06AM.log
Successfully Configured Software.
$

By now /etc/oratab will have +ASM entry added as the configuration is successful (Removed all comment lines in below o/p). You can also see that all the HAS resources are running as well

$ cat /etc/oratab
#Backup file is  /oracle/grid/crsdata/xxxxxxx/output/oratab.bak.xxxxxxx.oracle line added by Agent
# 
+ASM:/oracle/grid/19.0.0:N              # line added by Agent
$ 
$ . oraenv
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /oracle/grid
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.asm
               ONLINE  ONLINE       xxxxxxx                Started,STABLE
ora.ons
               OFFLINE OFFLINE      xxxxxxx                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       xxxxxxx                STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       xxxxxxx                STABLE
--------------------------------------------------------------------------------
$

We have successfully set up grid home now. All the below steps are optional and can be done according to your environment

Step 8: (Optional) Create required disk groups /alter diskgroup if any

Now we can alter the DATA diskgroup if it has more disks and also can create other disk groups such as TEMP, LOG, etc

Use asmca for creating disk groups as below

$ asmca -silent -createDiskGroup -diskGroupName TEMP -diskList /dev/oracleasm/TEMP1,/dev/oracleasm/TEMP2 -redundancy EXTERNAL -au_size 4

[DBT-30001] Disk groups created successfully. Check /oracle/grid/cfgtoollogs/asmca/asmca-200729AM105809.log for details.

$
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.LOG1.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.LOG2.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.RECO.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.TEMP.dg
               ONLINE  ONLINE       xxxxxxx                STABLE
ora.asm
               ONLINE  ONLINE       xxxxxxx                Started,STABLE
ora.ons
               OFFLINE OFFLINE      xxxxxxx                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       xxxxxxx                STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       xxxxxxx                STABLE
--------------------------------------------------------------------------------
$

Step 9: (Optional) Change compatibility of diskgroups

The database comapatibility of diskgroup added via asmca will be in 10.1 and can be set to a higher value if required accordingly

SQL> select group_number,name,compatibility,database_compatibility from v$asm_diskgroup;

GROUP_NUMBER NAME                           COMPATIBILITY        DATABASE_COMPATIBILI
------------ ------------------------------ -------------------- --------------------
           1 DATA                           19.0.0.0.0           12.1.0.0.0
           2 LOG1                           19.0.0.0.0           10.1.0.0.0
           3 LOG2                           19.0.0.0.0           10.1.0.0.0
           5 TEMP                           19.0.0.0.0           12.1.0.0.0
           4 RECO                           19.0.0.0.0           12.1.0.0.0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0
$ asmcmd setattr -G DATA compatible.rdbms 12.1.0.2.0
$ asmcmd setattr -G RECO compatible.rdbms 12.1.0.2.0
$ asmcmd setattr -G LOG1 compatible.rdbms 12.1.0.2.0
$ asmcmd setattr -G LOG2 compatible.rdbms 12.1.0.2.0
$ asmcmd setattr -G TEMP compatible.rdbms 12.1.0.2.0
$ sqlplus / as sysdba

SQL> select group_number,name,compatibility,database_compatibility from v$asm_diskgroup;

GROUP_NUMBER NAME                           COMPATIBILITY        DATABASE_COMPATIBILI
------------ ------------------------------ -------------------- --------------------
           1 DATA                           19.0.0.0.0           12.1.0.2.0
           2 LOG1                           19.0.0.0.0           12.1.0.2.0
           3 LOG2                           19.0.0.0.0           12.1.0.2.0
           5 TEMP                           19.0.0.0.0           12.1.0.2.0
           4 RECO                           19.0.0.0.0           12.1.0.2.0

SQL> exit
$

Now the grid home is ready for operation. We can see how to clone Oracle RDBMS home in my next post here

References: 

How to Clone an 11.2 Grid Infrastructure Home and Clusterware (Doc ID 1413846.1)
19.x:Clone.pl script is deprecated and how to clone using gold-image (Doc ID 2565006.1)

Happy cloning!!!

Enterprise Manager Cloud Control Agent deinstall steps

We have different methods where we can deinstall an OEM Agent. We will see the easiest way of them. First the graphical way of agent decommission introduced in version 12.1.0.4 and above and the silent method which is very easy to perform. The agent we are going to deinstall in this post of version 12.1.0.5. You can get the version as below and the Agent base location where all files reside

-sh-4.2$ /oracle/em_agent12/core/12.1.0.5.0/bin/emctl getversion agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
       Oracle Enterprise Manager Cloud Control Agent 12c Release 5
-sh-4.2$
-sh-4.2$
-sh-4.2$ pwd
/oracle/em_agent12
-sh-4.2$ ls -lrt
total 16
-rw-rw-r--. 1 oraoem dba  178 Jun 11  2015 agentimage.properties
-rwxr-xr-x. 1 oraoem dba  223 Nov 12  2018 plugins.txt
drwxr-xr-x. 3 oraoem dba   24 Nov 12  2018 core
-rw-r--r--. 1 oraoem dba  298 Nov 12  2018 plugins.txt.status
drwxr-xr-x. 8 oraoem dba 4096 Nov 13  2018 plugins
drwxr-xr-x. 5 oraoem dba  246 Nov  7  2019 sbin
drwxr-xr-x. 7 oraoem dba   74 Apr 11 03:42 agent_inst
-sh-4.2$

Decommmision Agent from OMS

First let’s see how to decommision the agent from within the OMS. Though the method is for 12c, it will be applicable to 13c OEM as well but with different page look.

Go to Setup =>> Manage Cloud Control =>> Agents

This would land you up in the below page where you are going to click on the server you need to decommission the agent.

I do click on the highlighted server (Name).

To decommission an agent the agent must be stopped. So once you land on the agent home page you can stop the agent by following the below.

Agent =>> Control =>> Shutdown

The next step is to provide the host credential to the server where you are trying to decommission the Agent. If you have already stored a named credential, you can choose it and proceed with clicking Shut Down button. You can also provide credentials (Username and Password) by clicking New credentials like I do as below

The following screens will appear to confirm agent shutdown.

You can then click on close and proceed to next step of decommission agent.

The following pics illustrate the Agent decommission steps

Agent =>> Target Setup =>> Click Agent Decommission... =>> Click Continue =>> Click Continue =>> Click Ok

Once the agent has been decommissioned, you will get the following confirmation in the OMS home page.

Decommmision Agent from OMS using command line

All the above can also be achieved by issuing commands in the OMS server which is the simplest way as you just need to find the agent from OMS and use emcli verbs to do the job as below.
Note: Agent on target host has to be down before we proceed with decommission

On Target Host: 

-sh-4.2$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
-sh-4.2$

On OMS server:

-sh-4.2$ emcli login -username=xxxx
Enter password :

Login successful
-sh-4.2$ emcli sync
Synchronized successfully
-sh-4.2$ emcli get_targets -target='oracle_emd' |grep xxxxx
4       Agent Unreachab  oracle_emd            xxxxx:3872
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli delete_target -name="xxxxx:3872" -type="oracle_emd" -delete_monitored_targets
Target "xxxxx:3872:oracle_emd" deleted successfully
-sh-4.2$

We can see that the agent is in "Agent Unreachable" state (line 8) as we already shutdown the agent in the target host.
Now use the delete_target verb (line 11) to delete all the monitored targets and to decommission the agent from the OMS.

Proceeding with Agent deinstall..

Now as we have decommissioned the Agent and its target from the OMS, we have 2 choice left

1. Either leave the agent software untouched so we can use it later if required or

2. Complete the deinstallation as below.

Steps are explained followed by actual commands in action.

1. Set AGENT home and AGENT instance home environment variables (lines 2,3)
2. List current plugins the agent installation contains and also create a plugin file which can be used later if you intend to reinstall the agent and use it with same or another OMS (lines 4, 15)
3. Make a copy of plugin file if you may need later (line 27)
4. Deinstall the agent using AgentDeinstall.pl script. (line 28)

-sh-4.2$
-sh-4.2$ export AGENT_HOME=/oracle/em_agent12/core/12.1.0.5.0
-sh-4.2$ export AGENT_INSTANCE_HOME=/oracle/em_agent12/agent_inst
-sh-4.2$ $AGENT_INSTANCE_HOME/bin/emctl listplugins agent -type all
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh    12.1.0.5.0           /oracle/em_agent12/plugins/oracle.sysman.oh.agent.plugin_12.1.0.5.0
oracle.sysman.db    12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.db.agent.plugin_12.1.0.8.0
oracle.sysman.xa    12.1.0.6.0           /oracle/em_agent12/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.6.0
oracle.sysman.db    12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.db.discovery.plugin_12.1.0.8.0
oracle.sysman.emas  12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.emas.discovery.plugin_12.1.0.8.0
oracle.sysman.oh    12.1.0.5.0           /oracle/em_agent12/plugins/oracle.sysman.oh.discovery.plugin_12.1.0.5.0

-sh-4.2$ $AGENT_HOME/perl/bin/perl $AGENT_HOME/sysman/install/create_plugin_list.pl -instancehome $AGENT_INSTANCE_HOME
The agent Base Dir is /oracle/em_agent12
 Creating file : /oracle/em_agent12/rwFile
Removed the /oracle/em_agent12/rwFile file.
oracle.sysman.xa|12.1.0.6.0||discoveryPlugin
oracle.sysman.db|12.1.0.8.0||discoveryPlugin
oracle.sysman.emas|12.1.0.8.0||discoveryPlugin
oracle.sysman.oh|12.1.0.5.0||discoveryPlugin
oracle.sysman.oh|12.1.0.5.0||agentPlugin
oracle.sysman.db|12.1.0.8.0||agentPlugin

/oracle/em_agent12/plugins.txt
-sh-4.2$ cp /oracle/em_agent12/plugins.txt /tmp/plugins.txt
-sh-4.2$ perl $AGENT_HOME/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME


 Agent Oracle Home: /oracle/em_agent12/core/12.1.0.5.0

agentHome = /oracle/em_agent12/core/12.1.0.5.0

NOTE: The agent base directory: /oracle/em_agent12 will be removed after successful deinstallation of agent home.


 DetachHome Command executed:/oracle/em_agent12/core/12.1.0.5.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/oracle/em_agent12/core/12.1.0.5.0 -waitForCompletion -invPtrLoc /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
The inventory pointer is located at /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
The inventory pointer is located at /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
The Oracle home '/oracle/em_agent12/sbin' could not be updated as it does not exist.


Deinstall Command executed:/oracle/em_agent12/core/12.1.0.5.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/oracle/em_agent12/core/12.1.0.5.0}" -waitForCompletion -removeAllFiles -invPtrLoc /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
Oracle Universal Installer, Version 11.1.0.13.0 Production
Copyright (C) 1999, 2015, Oracle. All rights reserved.

Starting deinstall


Deinstall in progress (Thursday, June 18, 2020 6:49:53 AM UTC)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Thursday, June 18, 2020 6:50:00 AM UTC)
End of deinstallations
Please check '/oracle/oraInventory/logs/silentInstall2020-06-18_06-49-50AM.log' for more details.

NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation.
-sh-4.2$
-sh-4.2$
-sh-4.2$
-sh-4.2$ pwd
/oracle/em_agent12
-sh-4.2$ ls -lrt
total 0
-sh-4.2$

Once after deinstall you can see that the Agent base directory is also cleaned up by the agent deinstall script.

Happy deinstalling..