Looking for something? Try here..

Tuesday, November 7, 2023

Oracle Enterprise Manager - undeploy deprecated plug-ins

 In my previous post, I explain on how to upgrade Oracle Enterprise Manager (OEM) from 13.4 to 13.5. In my lab environment, I have not installed any additional plug-ins. But this not the case when we did the same in live production environment. We had one plug-in which required to be migrated instead of upgraded as below. 


This is clearly indicated when we use GUI to perform the upgrade. If you are using command line to upgrade the OEM, then these prechecks are written to a file which gets buried in the huge number of lines as we tend to check only for any errors or warnings. This plug-in migration doesn't pop out as a warning or an error in the logs. 

The first attempt to upgrade resulted in OMS not starting up with incompatible plug-in version from OMS. The error matches with the symptoms indicated in support note EM 13c: Upgrade Fails At OMS startup With Error 'Plugin's Repository version is not compatible with Plugin's OMS version' (Doc ID 2491277.1)

Solution to the issue was to restore the old environment (gives you the reason why backing up the old home is very important), undeploy the deprecated plug-in on all agents and OMS and then perform the upgrade again. 

So where do we find which plug-ins are deprecated and how to undeploy them? 

You hit the following 2 support notes 

The check list note 13.5: Checklist for Upgrading Enterprise Manager Cloud Control from Version 13.3/13.4 to 13.5 (Doc ID 2761728.1) which talks about the obsolete plugins of 13.5 

The following plug-ins are obsolete and support for these plug-ins is discontinued. These plug-ins must be undeployed from the Oracle Management Agents and Oracle Management Services before proceeding with the upgrade.

  • Cloud Services Management (oracle.sysman.csm)
  • Microsoft .NET Framework (oracle.em.smdn)
  • Microsoft Active Directory (oracle.em.smad)

This note doesn't talk regarding the plug-in which was indicated in our prechecks or review page. 

The second note Oracle announces deprecation of Enterprise Manager 13c ORAchk Healthchecks Plug-in (Doc ID 2935520.1) gives exactly what we are searching for. 

So, we have the dedicated note for ORAchk Healthchecks plugin is deprecated in version 13.5. 

How do we check what other plugins are installed and whether we are good to proceed with the upgrade? 

Click on Setup => Plug-ins


This will list out all available plug-ins. 

Click on View => Only Deployed

This shows us all the deployed plug-ins in our environment. 


How to undeploy the plug-in? 

Click on the plug-in we need to undeploy => Undeploy From => Management servers... 


Since this plug-in is not installed in the Management Agents, the option is grayed out and only Management Servers is available to select. 


Input Sys Password and click Continue


Check the backup question and click Undeploy

If you haven't backed up the configuration, it can be done as below


Once we click on Undeploy, the plug-in will be undeloyed from Management server and we get the confirmation.


Progress can be seen by issuing emctl status oms -detail command


 Once completed click Close

Confirm the Deployed plug-ins 

We have successfully undeployed the deprecated plug-in and upgrade can be done without any issues. 

References: 

EM 13c: Upgrade Fails At OMS startup With Error 'Plugin's Repository version is not compatible with Plugin's OMS version' (Doc ID 2491277.1)
Undeploying Plug-Ins (oracle.com)

Happy Undeploying...!!! 

Wednesday, October 18, 2023

Oracle Enterprise Manager - 13.4 to 13.5 - Post upgrade

 In my previous post, we went through the steps of how to upgrade from OEM 13.4 to 13.5. This is the continuation of the post with all the post upgrade steps. Let's dive into action.. 

Note: If pictures look blurry in the post, please click on them to see the clear picture.

Post Upgrade: 

Modify the .bashrc or .profile file to reflect the new upgraded OMS home. We did an out of place upgrade and hence the path will be changing. Reflect the changes in the profile file to make our life easier. 

OMS will be brought up after the upgrade automatically once upgrade is done. Check for status of the OMS and EM KEY
[oracle@linux-8 ~]$ . ./.bashrc
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ echo $OMS_HOME
/oem/oem13.5
[oracle@linux-8 ~]$ emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
[oracle@linux-8 ~]$ emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
[oracle@linux-8 ~]$
Open the OEM console and check the version details.



Agent Checks:

Start the OEM agent which was stopped prior to upgrade and check whether all agents pushing data without issues and all targets status updated properly.
[oracle@linux-8 ~]$ /oem/oracle/agent/agent_13.4.0.0.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Starting agent ................... started.
[oracle@linux-8 ~]$
OEM Summary page will show the details of the targets at a glance. 


All target status uploaded without issues. Upgrade is completed successfully.

Next step will be to upgrade all the agents to 13.5.

Agent Patching: 

Prerequisites: 

1. Make sure Management agent is up and running, secured and not in black out state.

Check whether you have agent software available and applied in the software library. This is important if you want to upgrade agent on platform different than the OMS platform.

Perform following steps to check


Setup => Extensibility => Self Update => Agent Software


Check for Status = Applied

You can see since my OMS and target platforms are Linux x86-64, I'm good here. If you have any other platform such as windows, then you need to download the software and apply it on software library (steps not covered here). 

Check whether Agent is Up and Running and Secured

Setup => Manage Cloud control => Agents

Status should be Green and up
Secure Upload should be Yes.
If not up, start it up. If not secure, Click Secure to Secure the agent.  



2. Check for Not upgradable agents and fix them

Setup => Manage Cloud Control => Upgrade Agents => Not Upgradable Agents




You might see agents listed or sometimes not. Reason will be mentioned for the same. Fix it.
For the above issue, the solution is to run the below command from agent server

<EMSTATE>/bin/emctl control agent runCollection <TARGET_NAME>:oracle_home oracle_home_config

<EMSTATE> => Management Agent instance directory
<TARGET_NAME> => Management Agent home listed as oracle_home in the <EMSTATE>/sysman/emd/targets.xml file

or

Get the name from “emctl config agent listtargets |grep oracle_home” run from Agent home.

In my case,
<EMSTATE> = /oem/oracle/agent/agent_inst
<TARGET_NAME> = agent13c1_4_linux-8.selvapc.com_668
[oracle@linux-8 ~]$ /oem/oracle/agent/agent_13.4.0.0.0/bin/emctl config agent listtargets |grep oracle_home
[agent13c1_4_linux-8.selvapc.com_668, oracle_home]
[oms13c1_3_linux-8.selvapc.com_796, oracle_home]
[OraDB19Home1_2_linux-8.selvapc.com_5813, oracle_home]
[OraGI19Home1_1_linux-8.selvapc.com_8479, oracle_home]
[oms13c2_5_linux-8.selvapc.com_4905, oracle_home]
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ /oem/oracle/agent/agent_inst/bin/emctl control agent runCollection agent13c1_4_linux-8.selvapc.com_668:oracle_home oracle_home_config
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

[oracle@linux-8 ~]$
Now in the console, go to Management Agent Home page => Under Summary, Configuration => Click Oracle Home and Patch Details


Click Refresh Configuration




Re-verify Not Upgradable Agents


Now the issue is cleared. 

3. Other checks

Make sure the Management agent doesn’t come up during the upgrade process as cron or other schedule jobs might be running to bring up agent. Stop them.

Blackout creation is done during upgrade and is optional.

We are now set to upgrade the agent to 13.5

Upgrading Agent:

We can now upgrade the agents to version 13.5. Note this upgrade will only perform the upgrade to base version of the 13.5 agents. Once upgraded, patching to the agents needs to be done to bring it up to the latest RU. 

Setup => Manage Cloud Control => Upgrade Agents


Provide Job Name (or Accept default)
Click Add



Select all agents you want to upgrade and click OK
Fill in additional parameters if needed.  



Pre and post upgrade scripts to be run – provide path with name of script (should be placed in OMS server)
Additional parameters if any.
I leave everything default. (Check next pages on error encountered and fix to overcome)

Click Submit


Click OK
or you can use Override Privileged Credentials and provide credentials if you already have stored them.
I’m good with running root scripts on individual servers.  



Let’s wait for the upgrade to complete.
You can Click on individual agent targets to check on their respective progress as below.

 2 of my agent upgrade failed due to space issues in /tmp. At least 3 GB is required.  


I’ll now reinitiate for the failed ones but with different path for staging as below.  



Now all of my agents has been upgraded to version 13.5.0.0.0

Since we have not used privileged account, I have to run the root.sh script on all the agent servers to complete the agent upgrade.

Showing one server as below

[oracle@ol819-rac2 ~]$ sudo /oracle/agent/agent_13.5.0.0.0/root.sh

Finished product-specific root actions.
/etc exist
[oracle@ol819-rac2 ~]$
Now all the steps to upgrade the agent is completed. Let’s verify!

Setup => Manage Cloud Control => Agents


Click on any agent that you have upgraded of your choice



You can now see the version of the agent in the summary page.

If you would like to view the agent related details using EM CLI, following commands can be useful.

$<ORACLE_HOME>/bin/emcli get_agent_properties -format=csv -agent_name=<agent_host_name>:<agent_port>

$<ORACLE_HOME>/bin/emcli get_targets -format="name:csv" -targets=<agent_host_name>:<agent_port>:oracle_emd -alerts

Example given below to display contents on screen

emcli login -username=sysman
emcli sync
emcli get_agent_properties -agent_name=linux-8.selvapc.com:3872


To check status of all agent targets
emcli get_targets -targets=oracle_emd -alerts


Clean-up of Old Agents: 

Once the agents are updated to latest version, we can now clean up the old agents by performing below

Setup => Manage Cloud Control => Post Upgrade Tasks


Click Post Agent Upgrade Tasks => Add


Select all agents where you want to clean up old homes
Click OK


Click Submit



Click OK
Click Job Name to view progress


Clean up job succeeded and Old agent home is now removed.
Can be verified inside the server as well.

Before cleanup


After cleanup


We have now successfully completed the upgrade of OMS from 13.4 to 13.5 RU 16 and agents from 13.4 to 13.5. 
As I mentioned, agents need to be patched to latest RU. 

Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2) - We can download the latest available agent patch and apply. 

Happy Upgrading...!!!

Wednesday, October 11, 2023

Oracle Enterprise Manager - 13.4 to 13.5 upgrade

 In my previous post, we went through the steps of patching the Oracle Enterprise Manager (OEM) 13.4 from a lower patch level to a higher patch level 13.4.0.20. If you would like to stay in OEM 13.4, there is no more patches for the same. If you would like to upgrade to the terminal release of OEM 13c, let's check how that can be achieved. 

Note: If pictures look blurry in the post, please click on them to see the clear picture.

OEM 13.4 to 13.5 upgrade paths

There are 2 different path by which the OEM can be upgraded

1. Upgrade End to End

  • Takes long time 
  • Takes to base version and another downtime required to apply RU on 13.5
  • Needs OEM 13.4 to be version 13.4.0.10 or lesser patch level. 

2. Upgrade s/w only with plugins and configure later

  • Two phases - no downtime during 13.5 install resulting in lesser downtime for OEM
  • This path is recommended irrespective of the 13.4 patch level you are in. 
  • You should go straight to 13.5 RU01 or RU02 depending on version of patch if you are greater than 13.4.0.10. 

In this post, we will go through the steps of the second method mentioned. This post is going to be longer with multiple screenshots to explain all the steps. 

Main activities involve the below: 

  • Check list and completing prerequisites
  • Install OEM 13.5 out of place as a separate installation
  • Patching OEM 13.5 RU 16
  • Upgrade phase (Configure and upgrade 13.4 to 13.5) - Downtime
  • Post upgrade

Documents referenced:

  • OEM 13.4 to 13.5 - Upgrade Path (Doc ID 2815000.1)
  • 13.5: Checklist for Upgrading Enterprise Manager Cloud Control from Version 13.3/13.4 to 13.5 (Doc ID 2761728.1)
  • Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2)
Check List: 

A few items of the checklist can be checked before starting install of OMS 13.5 and a few can be verified before starting ConfigureGC.sh

Do not miss these prerequisite checklists to be completed as the upgrade cannot progress when these are satisfied.

1. Certified version of DB for Repository is 19c with RU11 or later. If you are on 12c versions, you will have to upgrade the repository DB to 19c with RU11 or later before the OMS upgrade. If your repository DB is an 18c, the upgrade will not be stopped, however, the DB should be upgraded immediately to 19c RU11 or later after the OMS upgrade.

2. Oracle recommends that you back up the Middleware Home, the Management Repository, Central inventory, Software library, Management Agent. Since the upgrade is performed Out of Place, the existing OMS Home and the Instance home are untouched. In case the upgrade fails, restore the repository DB from the backup taken prior to the upgrade.

3. Check if any agent running version < 13.3.0.0.0. If yes, upgrade them to 13.3.0.0.0 or later
SQL> select mt.target_name||' Agent Version='|| mtp.property_value from mgmt_target_properties mtp , mgmt_targets mt where mt.target_type='oracle_emd' and mt.target_guid = mtp.target_guid and mtp.property_name='Version' and mtp.property_value < '13.3.0.0.0';

no rows selected

SQL> 

4. Verify that all the OMS (including Standby) have the same version of Plugins deployed
SQL>    set serveroutput on
declare
oms_guids mgmt_medium_string_array := mgmt_medium_string_array();
oms_names mgmt_medium_string_array := mgmt_medium_string_array();
mismatches smp_emd_nvpair_array := smp_emd_nvpair_array();
begin
select host_url, em_plugin_util.get_guid_from_name(unique_oms_name)
bulk collect into oms_names, oms_guids
from gc_oms_info;
for j in 1..oms_guids.count
loop
dbms_output.put_line('Checking mismatch for OMS ' || oms_names(j));
mismatches := smp_emd_nvpair_array();
em_plugin_inventory.get_plugin_mismatch(oms_guids(j), mismatches);
if mismatches is null or mismatches.count = 0
then
dbms_output.put_line('No mismatch found for OMS ' || oms_names(j));
else
dbms_output.put_line('Mismatch found for OMS ' || oms_names(j));
for i in 1..mismatches.count
loop
dbms_output.put_line('Plug-in on repository: ' || nvl(mismatches(i).name, 'None') ||
', Plug-in on OMS: ' || nvl(mismatches(i).value, 'None'));
end loop;
end if;
end loop;
end;
/

Checking mismatch for OMS linux-8.selvapc.com:4889_Management_Service
No mismatch found for OMS linux-8.selvapc.com:4889_Management_Service

PL/SQL procedure successfully completed.

SQL>
If any of the following plugins are being used, download them () and stage them on the OMS_HOST.
Initiate the upgrade using: em13500_<platform>.bin PLUGIN_LOCATION=<path where the plugins are staged>

Plugins can be downloaded from below.
EM 13.5: https://www.oracle.com/enterprise-manager/downloads/oem-v135-update-plugins-downloads.html

Enterprise Manager for Times Ten
Enterprise Manager for People Soft
Enterprise Manager for Oracle Unified Directory
Enterprise Manager for Utilities (TUGBU)
Enterprise Manager for Golden Gate
Enterprise Manager for Database Appliance

Since I don't have these plugins, this step is not applicable to me. 

5. Check if large number of stuck jobs and clean up if present.
SQL> SELECT job_type, status, cnt
FROM (
SELECT job_type, status, count(*) as cnt
FROM MGMT_JOB_EXEC_SUMMARY join MGMT_JOB_TYPE_INFO using (job_type_id)
WHERE status NOT IN(3,4,5,8,18,19,23)
group by job_type, status
order by 3, 1, 2
)
where cnt > 10000;

no rows selected

SQL>
6. Ensure that the tables in the Management Repository do not have any snapshots created and drop if present.
SQL> select master, log_table from all_mview_logs where log_owner='SYSMAN';

no rows selected

SQL>

7. Ensure that theere are no logon or logoff triggers set in the Oracle Database that houses the Oracle Management Repository and disable them if any.
SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGON%' AND status='ENABLED';

no rows selected

SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';

TRIGGER_NAME
--------------------------------------------------------------------------------------------------------------------------------
GSMLOGOFF

SQL> select owner from dba_objects where object_name='GSMLOGOFF';

OWNER
--------------------------------------------------------------------------------------------------------------------------------
GSMADMIN_INTERNAL

SQL> alter trigger GSMADMIN_INTERNAL.GSMLOGOFF disable;
alter trigger GSMADMIN_INTERNAL.GSMLOGOFF disable
*
ERROR at line 1:
ORA-01031: insufficient privileges


SQL> conn sys@rmancat as sysdba
Enter password:
Connected.
SQL> alter trigger GSMADMIN_INTERNAL.GSMLOGOFF disable;

Trigger altered.

SQL>
rmancat is the connect identifier to my OEM repository PDB database. 

8. Disable the password_verify_function on the repository DB

9. Make sure no active sysman sessions are running in the Repository database. This is needed before downtime of the OMS
SQL> col username for a30
SQL> col action for a10
SQL> col machine for a30
SQL> select sid,serial#,username,status,action,machine from v$session where username is not null and status='ACTIVE';

       SID    SERIAL# USERNAME                       STATUS   ACTION     MACHINE
---------- ---------- ------------------------------ -------- ---------- ------------------------------
         2      31258 SYS                            ACTIVE              linux-8.selvapc.com
       115      41695 SYS                            ACTIVE              linux-8.selvapc.com

SQL>
10. Following init parameters should be set/reset prior to upgraded. If these parameters are not set/reset, then prerequisites will fail and won’t be able to progress.

Note: The parameters need to be set at CDB level if you are using multitenant architecture and repository database is a PDB.
SQL> alter system set "_allow_insert_with_update_check"=true scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';

System altered.

SQL> alter system reset "_px_adaptive_dist_method" scope=both sid='*';

System altered.

SQL> alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_use_feedback" scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_gather_feedback" scope=both sid='*';

System altered.

SQL> alter system reset "_optimizer_performance_feedback" scope=both sid='*';

System altered.

SQL>
11. select username from dba_users where username like '%OPSS%';

1) If SYSMANUPGR_OPSS is present, run the query and skip the second step:

select * from sysmanUPGR_opss.jps_dn where parentdn like '%cn=roles%' and (rdn like '%'||CHR(9)||'%' OR rdn like '%\%' OR rdn like ' %' OR rdn like '% ') and (rdn like '%cn=%')

2) If SYSMAN122130_OPSS is present, run the query:

select * from sysman122130_opss.jps_dn where parentdn like '%cn=roles%' and (rdn like '%'||CHR(9)||'%' OR rdn like '%\%' OR rdn like ' %' OR rdn like '% ') and (rdn like '%cn=%')

If this query returns any rows, contact Oracle Support. If no rows are returned, continue with the upgrade.
SQL> select username from dba_users where username like '%OPSS%';

USERNAME
------------------------------
SYSMAN122130_OPSS

SQL> select * from sysman122130_opss.jps_dn where parentdn like '%cn=roles%' and (rdn like '%'||CHR(9)||'%' OR rdn like '%\%' OR rdn like ' %' OR rdn like '% ') and (rdn like '%cn=%');

no rows selected

SQL>
12. Copy the EM KEY into repository.
Issue below command to copy the EM KEY into the repository
[oracle@linux-8 ~]$ /oem/oracle/middleware/bin/emctl config emkey -copy_to_repos -sysman_pwd Welcome123
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ /oem/oracle/middleware/bin/emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey  is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos".
[oracle@linux-8 ~]$
All the checklist has now been completed to start the OEM 13.5 upgrade. 

Install OEM 13.5

Once all the above checklist is completed (or which ever requires downtime can wait until downtime window), let’s start with OEM 13.5 installation. This upgrade will be out of place upgrade and is recommended method as if something goes wrong during upgrade, we still have the OEM 13.4 untouched.
[oracle@linux-8 ~]$ cd /media/sf_Oracle/software/OEM13.5/
[oracle@linux-8 OEM13.5]$ ls -lrt
total 9534476
-rwxrwx---. 1 root vboxsf 1897494675 Sep  1 10:08 em13500_linux64-2.zip
-rwxrwx---. 1 root vboxsf 2130750885 Sep  1 10:11 em13500_linux64-3.zip
-rwxrwx---. 1 root vboxsf 1454982045 Sep  1 10:13 em13500_linux64-4.zip
-rwxrwx---. 1 root vboxsf 1494500710 Sep  1 10:15 em13500_linux64-5.zip
-rwxrwx---. 1 root vboxsf 1586801533 Sep  1 10:15 em13500_linux64.bin
-rwxrwx---. 1 root vboxsf 1198762103 Sep  5 22:06 p35437906_135000_Generic.zip
[oracle@linux-8 OEM13.5]$ 
[oracle@linux-8 OEM13.5]$ ./em13500_linux64.bin ENABLE_SSL=true
ERROR: Temporary directory /tmp does not have enough free space. At least 12289 MB of free space are required.
Please input another directory or [Exit]: /media/sf_Oracle/software/OEM13.5
Launcher log file is /media/sf_Oracle/software/OEM13.5/OraInstall2023-09-05_11-31-58PM/launcher2023-09-05_11-31-58PM.log.
Extracting the installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  . . . . . . . . . . . . . . . . . . Done
Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed
Checking swap space: must be greater than 512 MB.   Actual 4095 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Preparing to launch the Oracle Universal Installer from /media/sf_Oracle/software/OEM13.5/OraInstall2023-09-05_11-31-58PM
ScratchPathValue :/media/sf_Oracle/software/OEM13.5/OraInstall2023-09-05_11-31-58PM
Sep 05, 2023 11:35:03 PM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider
INFO: No detected/configured IoServiceFactoryFactory using Nio2ServiceFactoryFactory
...
...


Click Next


Provide the path where you have your RU downloaded and unzipped.
This patch is needed only when you have oneoff patches to update and not for RU. For RU update, we do this post installation.
Click Next


Click Next


Since this is just installation, a few checks won’t be executed but these will be done during upgrade and if not satisfied, the upgrade won’t continue.

Click on Yes


Provide new middleware home and click Next


Connect Descriptor will be auto picked up. Verify it. Non editable field
Enter sys and sysman password.
Since I don’t have SSL configured for my DB, I don’t check that box.

Click Next


I don’t have any failures in prerequisites. Hence continuing
Warnings are ok, but check if these can be fixed and fix it if possible.

Click Next


Information page and my environment: Differs with different environment.

1. Mandatory patch

If your repository database is on 19.18, Apply Patch 35046819 to the repository database before applying OMS Release Update Patch. Since my repository DB is version 19.20, this can be skipped

2. Running emctl status oms -details gives me the below

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: linux-8.selvapc.com
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up

[oracle@linux-8 OEM13.5]$ emctl extended oms jvmd stop -all
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved.
No verb found to be registered with emctl extensibles framework
[oracle@linux-8 OEM13.5]$

EM 13c :OEM upgrade is Failing With Error :"No verb found to be registered with emctl extensibles framework while stopping the jvmd" (Doc ID 2809633.1)

Solution: Ignore this message for now and proceed.

3. In real time environment, we can create black out for all the targets. For now, I’ll stop agent

[oracle@linux-8 OEM13.5]$ /oem/oracle/agent/agent_13.4.0.0.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved.

Stopping agent ... stopped.
[oracle@linux-8 OEM13.5]$


4. If BI Publisher report is needed, then OAS needs to be installed. Otherwise we are good here.

5. Let’s apply the latest patch once installation is done.

Click Ok
Click Next


If you have any of the plugins installed, please select to upgrade. I have default selection in my 13.4 and hence I don’t select any plugin here.

Click Next


Review the installation location and click Upgrade.

 


[oracle@linux-8 OEM13.5]$ sudo su -
[sudo] password for oracle:
[root@linux-8 ~]# /oem/oem13.5/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /oem/oem13.5/root.sh ......
Check /oem/oem13.5/install/root_linux-8.selvapc.com_2023-09-06_01-47-39.log for the output of root script
Finished execution of  /oem/oem13.5/root.sh ......


[root@linux-8 ~]#
Run the script as root user and click OK


Installation of OEM 13.5 is now complete.

At this point OEM (OMS) 13.4 is still up and running. We can start the agent now or leave till we complete upgrade.


Patching OEM 13.5 RU 16

Let’s patch the OEM 13.5 with RU 16 which is the latest available RU at the time of writing this post. 

Referenced document:
  • 13.5: How to Apply Release Update on the OMS During the Install/Upgrade (Doc ID 2810169.1)
  • Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2)
Patches needed:
  • Download Patch 19999993: OMSPatcher patch of version 13.9.5.14.0 for Enterprise Manager Cloud Control 13.5.0.0.0
  • Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2) - p35437906_135000_Generic.zip
Prerequisites:

Update omspatcher of OEM13.5 Home

Current version: 
[oracle@linux-8 OEM13.5]$ cd /oem/oem13.5/OMSPatcher/
[oracle@linux-8 OMSPatcher]$ ./omspatcher version
OMSPatcher Version: 13.9.4.4.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020

OMSPatcher succeeded.
[oracle@linux-8 OMSPatcher]$
Steps:

1. Download Patch 19999993: OMSPatcher patch of version 13.9.5.14.0 for Enterprise Manager Cloud Control 13.5.0.0.0
2. mv current OMSPatcher
3. unzip patch file under /oem/oem13.5/
4. Check version
[oracle@linux-8 OMSPatcher]$ cd ..
[oracle@linux-8 oem13.5]$ mv OMSPatcher OMSPatcher_old
[oracle@linux-8 oem13.5]$ unzip /media/sf_Oracle/software/OEM13.5/p19999993_135000_Generic.zip
[oracle@linux-8 oem13.5]$ cd OMSPatcher
[oracle@linux-8 OMSPatcher]$ ./omspatcher version
OMSPatcher Version: 13.9.5.14.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020

OMSPatcher succeeded.
[oracle@linux-8 OMSPatcher]$
Once the prerequisite is complete, apply the patch OEM 13.5 RU 16 as below

Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2)

Download p35437906_135000_Generic.zip from above note and unzip to staging location and apply patch using -bitonly option
[oracle@linux-8 OEM13.5]$ ls -lrt |head -3
total 9535266
drwxrwx---. 1 root vboxsf       4096 Jun 19 12:21 35437906
-rwxrwx---. 1 root vboxsf       2368 Jul 12 16:32 PatchSearch.xml
[oracle@linux-8 OEM13.5]$
[oracle@linux-8 OEM13.5]$ cd 35437906/
[oracle@linux-8 35437906]$ export ORACLE_HOME=/oem/oem13.5
Analyze the patch and if all is good, Apply! 

[oracle@linux-8 35437906]$ $ORACLE_HOME/OMSPatcher/omspatcher apply -analyze -bitonly
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.14.0
OUI version        : 13.9.4.0.0
Running from       : /oem/oem13.5
Log file location  : /oem/oem13.5/cfgtoollogs/omspatcher/opatch2023-09-06_02-25-50AM_1.log

OMSPatcher log file: /oem/oem13.5/cfgtoollogs/omspatcher/35437906/omspatcher_2023-09-06_02-25-53AM_analyze.log

WARNING: OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.

Do you want to proceed? [y|n]
y
User Responded with: Y


Prereq "checkComponents" for patch 35258570 passed.

Prereq "checkComponents" for patch 34601404 passed.

Prereq "checkComponents" for patch 35122519 passed.

Prereq "checkComponents" for patch 35258589 passed.

Prereq "checkComponents" for patch 35258598 passed.

Prereq "checkComponents" for patch 35258702 passed.

Running apply prerequisite checks for sub-patch(es) "35258598,35258570,35258589,35258702,35122519,34601404" and Oracle Home "/oem/oem13.5"...
Sub-patch(es) "35258598,35258570,35258589,35258702,35122519,34601404" are successfully analyzed for Oracle Home "/oem/oem13.5"


Complete Summary
================


All log file names referenced below can be accessed from the directory "/oem/oem13.5/cfgtoollogs/omspatcher/2023-09-06_02-25-50AM_SystemPatch_35437906_1"

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

             Featureset                                             Sub-patches                                                                                  Log file
             ----------                                             -----------                                                                                  --------
  oracle.sysman.top.oms   35258598,35258570,35258589,35258702,35122519,34601404   35258598,35258570,35258589,35258702,35122519,34601404_opatch2023-09-06_02-25-57AM_1.log


The following sub-patches are incompatible with components installed in the OMS system:
34430473,34430509,34706773,35258654,34706739,35258672,35122483,35258713,35122522



--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.
--------------------------------------------------------------------------------
Log file location: /oem/oem13.5/cfgtoollogs/omspatcher/35437906/omspatcher_2023-09-06_02-25-53AM_analyze.log

OMSPatcher succeeded.
[oracle@linux-8 35437906]$

[oracle@linux-8 35437906]$ $ORACLE_HOME/OMSPatcher/omspatcher apply -bitonly
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.14.0
OUI version        : 13.9.4.0.0
Running from       : /oem/oem13.5
Log file location  : /oem/oem13.5/cfgtoollogs/omspatcher/opatch2023-09-06_02-28-42AM_1.log

WARNING: OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.

Do you want to proceed? [y|n]
y
User Responded with: Y


Prereq "checkComponents" for patch 35258570 passed.
…
…
…
To continue, OMSPatcher will do the following:
[Patch and deploy artifacts]   :


Do you want to proceed? [y|n]
y
User Responded with: Y
…
…
…
The following sub-patches are incompatible with components installed in the OMS system:
34430473,34430509,34706773,35258654,34706739,35258672,35122483,35258713,35122522



--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.
--------------------------------------------------------------------------------
Log file location: /oem/oem13.5/cfgtoollogs/omspatcher/35437906/omspatcher_2023-09-06_02-28-47AM_apply.log

OMSPatcher succeeded.
[oracle@linux-8 35437906]$
[oracle@linux-8 35437906]$ $ORACLE_HOME/OMSPatcher/omspatcher lspatches | grep oms
Log file location  : /oem/oem13.5/cfgtoollogs/omspatcher/opatch2023-09-06_02-35-43AM_1.log
oracle.sysman.db.oms.plugin/13.5.1.0.0            Plugin              35437906            35258598            Oracle Enterprise Manager for Oracle Database 13c Release 5 Plug-in Update 16 (13.5.1.16) for Oracle Management Service
oracle.sysman.cfw.oms.plugin/13.5.1.0.0           Plugin              35437906            34601404            Oracle Enterprise Manager for Cloud Framework 13c Release 5 Plug-in Update 11 (13.5.1.11) for Oracle Management Service
oracle.sysman.xa.oms.plugin/13.5.1.0.0            Plugin              35437906            35258702            Oracle Enterprise Manager for Exadata 13c Release 5 Plug-in Update 16 (13.5.1.16) for Oracle Management Service
oracle.sysman.top.oms/13.5.0.0.0                  Core                35437906            35258570            Oracle Enterprise Manager 13c Release 5 Platform Update 16 (13.5.0.16) for Oracle Management Service
oracle.sysman.si.oms.plugin/13.5.1.0.0            Plugin              35437906            35122519            Oracle Enterprise Manager for Systems Infrastructure 13c Release 5 Plug-in Update 15 (13.5.1.15) for Oracle Management Service
oracle.sysman.emas.oms.plugin/13.5.1.0.0          Plugin              35437906            35258589            Oracle Enterprise Manager for Fusion Middleware 13c Release 5 Plug-in Update 16 (13.5.1.16) for Oracle Management Service
[oracle@linux-8 35437906]$

Patching OEM 13.5 to latest version is now completed.

All the OEM 13.4 components are still available until this point.

Upgrade phase:

Let’s now upgrade the OMS 13.4 to OMS 13.5 This is where the downtime for OMS starts.

Shut the OMS 13.4. Shut agent 13.4, if running.
[oracle@linux-8 OMSPatcher]$ echo $OMS_HOME
/oem/oracle/middleware
[oracle@linux-8 OMSPatcher]$ echo $PATH
/home/oracle/.local/bin:/home/oracle/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/oracle/db/db19/bin:/oem/oracle/middleware/bin
[oracle@linux-8 OMSPatcher]$
[oracle@linux-8 OMSPatcher]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
[oracle@linux-8 OMSPatcher]$

Start configureGC.sh to configure and upgrade the OMS 13.4 to 13.5

[oracle@linux-8 ~]$ export ORACLE_HOME=/oem/oem13.5
[oracle@linux-8 ~]$ $ORACLE_HOME/sysman/install/ConfigureGC.sh START_OMS=true
ScratchPathValue :/oem/oem13.5
*************************************
Error Messages:
----------------------------------------------------
* Invalid option(s): sourceLoc
Sep 06, 2023 9:40:07 AM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider
INFO: No detected/configured IoServiceFactoryFactory using Nio2ServiceFactoryFactory
List file is unreadable


Choose the installation type and current Home
Click Next


Verify Connect Descriptor – non editable field
Provide sys and sysman password
Since I don’t have SSL configured for my repository DB, I don’t select the option.

Click Next


If all the prerequisites mentioned in the Prerequisites section is not properly completed, the following screen might appear in which we can’t proceed.


Complete all the prerequisites. This is listed in the checklist section of this document.
For Auto fixable => click Auto fix

Click Check Again


Take a look at the warnings. Since my env is my home lab, I’m ignoring these warnings and proceeding. In Prod environment, make sure we don’t have any warnings as well since these can cause performance issues or other undesirable issues later on.

Click Next


Provide Weblogic user password. This is the existing password of OMS 13.4
Provide OMS instance base location outside of new OMS Home. Accept if default is ok

Click Next


Verify the Port configuration. Change if needed. I left it at default ports.

Click Next


Review the details
Click Configure


Configuration and upgrade is in progress... 

In progress...

 In progress… A lot happens in the back ground as well.  




Upgrade is completed.

Click Close

Open the OEM console and check the version details.

We have now successfully completed the OMS upgrade from 13.4 to 13.5 RU 16. 

What are the post upgrade activities which needs to be taken care? Let's look at them in the next post. 

References: 

All the reference articles are provided in their respective sections. 

Happy Upgrading...!!!