Oracle Enterprise Manager - 13.4 to 13.5 - Post upgrade

 In my previous post, we went through the steps of how to upgrade from OEM 13.4 to 13.5. This is the continuation of the post with all the post upgrade steps. Let's dive into action.. 

Note: If pictures look blurry in the post, please click on them to see the clear picture.

Post Upgrade: 

Modify the .bashrc or .profile file to reflect the new upgraded OMS home. We did an out of place upgrade and hence the path will be changing. Reflect the changes in the profile file to make our life easier. 

OMS will be brought up after the upgrade automatically once upgrade is done. Check for status of the OMS and EM KEY

[oracle@linux-8 ~]$ . ./.bashrc
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ echo $OMS_HOME
/oem/oem13.5
[oracle@linux-8 ~]$ emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
[oracle@linux-8 ~]$ emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
[oracle@linux-8 ~]$

Open the OEM console and check the version details.

Agent Checks:

Start the OEM agent which was stopped prior to upgrade and check whether all agents pushing data without issues and all targets status updated properly.

[oracle@linux-8 ~]$ /oem/oracle/agent/agent_13.4.0.0.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Starting agent ................... started.
[oracle@linux-8 ~]$

OEM Summary page will show the details of the targets at a glance. 

All target status uploaded without issues. Upgrade is completed successfully.

Next step will be to upgrade all the agents to 13.5.

Agent Patching: 

Prerequisites: 

1. Make sure Management agent is up and running, secured and not in black out state.

Check whether you have agent software available and applied in the software library. This is important if you want to upgrade agent on platform different than the OMS platform.

Perform following steps to check

Setup => Extensibility => Self Update => Agent Software

Check for Status = Applied

You can see since my OMS and target platforms are Linux x86-64, I'm good here. If you have any other platform such as windows, then you need to download the software and apply it on software library (steps not covered here). 

Check whether Agent is Up and Running and Secured

Setup => Manage Cloud control => Agents

Status should be Green and up
Secure Upload should be Yes.
If not up, start it up. If not secure, Click Secure to Secure the agent.  

2. Check for Not upgradable agents and fix them

Setup => Manage Cloud Control => Upgrade Agents => Not Upgradable Agents

You might see agents listed or sometimes not. Reason will be mentioned for the same. Fix it.
For the above issue, the solution is to run the below command from agent server

<EMSTATE>/bin/emctl control agent runCollection <TARGET_NAME>:oracle_home oracle_home_config

<EMSTATE> => Management Agent instance directory
<TARGET_NAME> => Management Agent home listed as oracle_home in the <EMSTATE>/sysman/emd/targets.xml file

or

Get the name from “emctl config agent listtargets |grep oracle_home” run from Agent home.

In my case,
<EMSTATE> = /oem/oracle/agent/agent_inst
<TARGET_NAME> = agent13c1_4_linux-8.selvapc.com_668

[oracle@linux-8 ~]$ /oem/oracle/agent/agent_13.4.0.0.0/bin/emctl config agent listtargets |grep oracle_home
[agent13c1_4_linux-8.selvapc.com_668, oracle_home]
[oms13c1_3_linux-8.selvapc.com_796, oracle_home]
[OraDB19Home1_2_linux-8.selvapc.com_5813, oracle_home]
[OraGI19Home1_1_linux-8.selvapc.com_8479, oracle_home]
[oms13c2_5_linux-8.selvapc.com_4905, oracle_home]
[oracle@linux-8 ~]$
[oracle@linux-8 ~]$ /oem/oracle/agent/agent_inst/bin/emctl control agent runCollection agent13c1_4_linux-8.selvapc.com_668:oracle_home oracle_home_config
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

[oracle@linux-8 ~]$

Now in the console, go to Management Agent Home page => Under Summary, Configuration => Click Oracle Home and Patch Details

Click Refresh Configuration

Re-verify Not Upgradable Agents

Now the issue is cleared. 

3. Other checks

Make sure the Management agent doesn’t come up during the upgrade process as cron or other schedule jobs might be running to bring up agent. Stop them.

Blackout creation is done during upgrade and is optional.

We are now set to upgrade the agent to 13.5

Upgrading Agent:

We can now upgrade the agents to version 13.5. Note this upgrade will only perform the upgrade to base version of the 13.5 agents. Once upgraded, patching to the agents needs to be done to bring it up to the latest RU. 

Setup => Manage Cloud Control => Upgrade Agents

Provide Job Name (or Accept default)
Click Add

Select all agents you want to upgrade and click OK
Fill in additional parameters if needed.  

Pre and post upgrade scripts to be run – provide path with name of script (should be placed in OMS server)
Additional parameters if any.
I leave everything default. (Check next pages on error encountered and fix to overcome)

Click Submit

Click OK
or you can use Override Privileged Credentials and provide credentials if you already have stored them.
I’m good with running root scripts on individual servers.  

Let’s wait for the upgrade to complete.
You can Click on individual agent targets to check on their respective progress as below.

 2 of my agent upgrade failed due to space issues in /tmp. At least 3 GB is required.  

I’ll now reinitiate for the failed ones but with different path for staging as below.  

Now all of my agents has been upgraded to version 13.5.0.0.0

Since we have not used privileged account, I have to run the root.sh script on all the agent servers to complete the agent upgrade.

Showing one server as below

[oracle@ol819-rac2 ~]$ sudo /oracle/agent/agent_13.5.0.0.0/root.sh

Finished product-specific root actions.
/etc exist
[oracle@ol819-rac2 ~]$

Now all the steps to upgrade the agent is completed. Let’s verify!

Setup => Manage Cloud Control => Agents

Click on any agent that you have upgraded of your choice

You can now see the version of the agent in the summary page.

If you would like to view the agent related details using EM CLI, following commands can be useful.

$<ORACLE_HOME>/bin/emcli get_agent_properties -format=csv -agent_name=<agent_host_name>:<agent_port>

$<ORACLE_HOME>/bin/emcli get_targets -format="name:csv" -targets=<agent_host_name>:<agent_port>:oracle_emd -alerts

Example given below to display contents on screen

emcli login -username=sysman
emcli sync
emcli get_agent_properties -agent_name=linux-8.selvapc.com:3872

To check status of all agent targets
emcli get_targets -targets=oracle_emd -alerts

Clean-up of Old Agents: 

Once the agents are updated to latest version, we can now clean up the old agents by performing below

Setup => Manage Cloud Control => Post Upgrade Tasks

Click Post Agent Upgrade Tasks => Add

Select all agents where you want to clean up old homes
Click OK

Click Submit

Click OK

Click Job Name to view progress

Clean up job succeeded and Old agent home is now removed.
Can be verified inside the server as well.

Before cleanup

After cleanup

We have now successfully completed the upgrade of OMS from 13.4 to 13.5 RU 16 and agents from 13.4 to 13.5. 

As I mentioned, agents need to be patched to latest RU. 

Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2) - We can download the latest available agent patch and apply. 

Happy Upgrading...!!!