Thursday, June 18, 2020

Enterprise Manager Cloud Control Agent deinstall steps

We have different methods where we can deinstall an OEM Agent. We will see the easiest way of them. First the graphical way of agent decommission introduced in version 12.1.0.4 and above and the silent method which is very easy to perform. The agent we are going to deinstall in this post of version 12.1.0.5. You can get the version as below and the Agent base location where all files reside
-sh-4.2$ /oracle/em_agent12/core/12.1.0.5.0/bin/emctl getversion agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
       Oracle Enterprise Manager Cloud Control Agent 12c Release 5
-sh-4.2$
-sh-4.2$
-sh-4.2$ pwd
/oracle/em_agent12
-sh-4.2$ ls -lrt
total 16
-rw-rw-r--. 1 oraoem dba  178 Jun 11  2015 agentimage.properties
-rwxr-xr-x. 1 oraoem dba  223 Nov 12  2018 plugins.txt
drwxr-xr-x. 3 oraoem dba   24 Nov 12  2018 core
-rw-r--r--. 1 oraoem dba  298 Nov 12  2018 plugins.txt.status
drwxr-xr-x. 8 oraoem dba 4096 Nov 13  2018 plugins
drwxr-xr-x. 5 oraoem dba  246 Nov  7  2019 sbin
drwxr-xr-x. 7 oraoem dba   74 Apr 11 03:42 agent_inst
-sh-4.2$

Decommmision Agent from OMS

First let’s see how to decommision the agent from within the OMS. Though the method is for 12c, it will be applicable to 13c OEM as well but with different page look.

Go to Setup =>> Manage Cloud Control =>> Agents


This would land you up in the below page where you are going to click on the server you need to decommission the agent.


I do click on the highlighted server (Name).

To decommission an agent the agent must be stopped. So once you land on the agent home page you can stop the agent by following the below.

Agent =>> Control =>> Shutdown


The next step is to provide the host credential to the server where you are trying to decommission the Agent. If you have already stored a named credential, you can choose it and proceed with clicking Shut Down button. You can also provide credentials (Username and Password) by clicking New credentials like I do as below

The following screens will appear to confirm agent shutdown.


You can then click on close and proceed to next step of decommission agent.

The following pics illustrate the Agent decommission steps
Agent =>> Target Setup =>> Click Agent Decommission... =>> Click Continue =>> Click Continue =>> Click Ok



Once the agent has been decommissioned, you will get the following confirmation in the OMS home page.


Decommmision Agent from OMS using command line

All the above can also be achieved by issuing commands in the OMS server which is the simplest way as you just need to find the agent from OMS and use emcli verbs to do the job as below.
Note: Agent on target host has to be down before we proceed with decommission

On Target Host: 
-sh-4.2$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
-sh-4.2$
On OMS server:
-sh-4.2$ emcli login -username=xxxx
Enter password :

Login successful
-sh-4.2$ emcli sync
Synchronized successfully
-sh-4.2$ emcli get_targets -target='oracle_emd' |grep xxxxx
4       Agent Unreachab  oracle_emd            xxxxx:3872
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli delete_target -name="xxxxx:3872" -type="oracle_emd" -delete_monitored_targets
Target "xxxxx:3872:oracle_emd" deleted successfully
-sh-4.2$
We can see that the agent is in "Agent Unreachable" state (line 8) as we already shutdown the agent in the target host.
Now use the delete_target verb (line 11) to delete all the monitored targets and to decommission the agent from the OMS.

Proceeding with Agent deinstall..

Now as we have decommissioned the Agent and its target from the OMS, we have 2 choice left
1. Either leave the agent software untouched so we can use it later if required or
2. Complete the deinstallation as below.

Steps are explained followed by actual commands in action.

1. Set AGENT home and AGENT instance home environment variables (lines 2,3)
2. List current plugins the agent installation contains and also create a plugin file which can be used later if you intend to reinstall the agent and use it with same or another OMS (lines 4, 15)
3. Make a copy of plugin file if you may need later (line 27)
4. Deinstall the agent using AgentDeinstall.pl script. (line 28)
-sh-4.2$
-sh-4.2$ export AGENT_HOME=/oracle/em_agent12/core/12.1.0.5.0
-sh-4.2$ export AGENT_INSTANCE_HOME=/oracle/em_agent12/agent_inst
-sh-4.2$ $AGENT_INSTANCE_HOME/bin/emctl listplugins agent -type all
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
oracle.sysman.oh    12.1.0.5.0           /oracle/em_agent12/plugins/oracle.sysman.oh.agent.plugin_12.1.0.5.0
oracle.sysman.db    12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.db.agent.plugin_12.1.0.8.0
oracle.sysman.xa    12.1.0.6.0           /oracle/em_agent12/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.6.0
oracle.sysman.db    12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.db.discovery.plugin_12.1.0.8.0
oracle.sysman.emas  12.1.0.8.0           /oracle/em_agent12/plugins/oracle.sysman.emas.discovery.plugin_12.1.0.8.0
oracle.sysman.oh    12.1.0.5.0           /oracle/em_agent12/plugins/oracle.sysman.oh.discovery.plugin_12.1.0.5.0

-sh-4.2$ $AGENT_HOME/perl/bin/perl $AGENT_HOME/sysman/install/create_plugin_list.pl -instancehome $AGENT_INSTANCE_HOME
The agent Base Dir is /oracle/em_agent12
 Creating file : /oracle/em_agent12/rwFile
Removed the /oracle/em_agent12/rwFile file.
oracle.sysman.xa|12.1.0.6.0||discoveryPlugin
oracle.sysman.db|12.1.0.8.0||discoveryPlugin
oracle.sysman.emas|12.1.0.8.0||discoveryPlugin
oracle.sysman.oh|12.1.0.5.0||discoveryPlugin
oracle.sysman.oh|12.1.0.5.0||agentPlugin
oracle.sysman.db|12.1.0.8.0||agentPlugin

/oracle/em_agent12/plugins.txt
-sh-4.2$ cp /oracle/em_agent12/plugins.txt /tmp/plugins.txt
-sh-4.2$ perl $AGENT_HOME/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME


 Agent Oracle Home: /oracle/em_agent12/core/12.1.0.5.0

agentHome = /oracle/em_agent12/core/12.1.0.5.0

NOTE: The agent base directory: /oracle/em_agent12 will be removed after successful deinstallation of agent home.


 DetachHome Command executed:/oracle/em_agent12/core/12.1.0.5.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/oracle/em_agent12/core/12.1.0.5.0 -waitForCompletion -invPtrLoc /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
The inventory pointer is located at /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
The inventory pointer is located at /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
The Oracle home '/oracle/em_agent12/sbin' could not be updated as it does not exist.


Deinstall Command executed:/oracle/em_agent12/core/12.1.0.5.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/oracle/em_agent12/core/12.1.0.5.0}" -waitForCompletion -removeAllFiles -invPtrLoc /oracle/em_agent12/core/12.1.0.5.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 36799 MB    Passed
Oracle Universal Installer, Version 11.1.0.13.0 Production
Copyright (C) 1999, 2015, Oracle. All rights reserved.

Starting deinstall


Deinstall in progress (Thursday, June 18, 2020 6:49:53 AM UTC)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Thursday, June 18, 2020 6:50:00 AM UTC)
End of deinstallations
Please check '/oracle/oraInventory/logs/silentInstall2020-06-18_06-49-50AM.log' for more details.

NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation.
-sh-4.2$
-sh-4.2$
-sh-4.2$
-sh-4.2$ pwd
/oracle/em_agent12
-sh-4.2$ ls -lrt
total 0
-sh-4.2$
Once after deinstall you can see that the Agent base directory is also cleaned up by the agent deinstall script.

Happy deinstalling.. 

Tuesday, June 9, 2020

PASSWORDFILE_METADATA_CACHE

Environment details: 
Oracle Database Version 19.6.0.0.0 running on EL7

Issue: 
Connection to sys as sysdba via network is throwing invalid username/password error
[oracle@xxxx admin]$ sqlplus sys@abcdb:1522/+ASM as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 9 02:30:40 2020
Version 19.6.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Enter password:
ERROR:
ORA-01017: invalid username/password; logon denied 
I have to tried changing the password and still the error ORA-01017 occur.
I checked the password related parameter remote_login_passwordfile and it is set to EXCLUSIVE
Checked the password file whether it is present or not.
[oracle@xxxx ~]$ asmcmd pwget --asm
+ABC_DATA_DG01/orapwasm

Cause:
This is the issue in my environment. We have recently changed the disk group names from +ABC_DATA_DG01 to +DATA as part of standardization. So the password file name didn't change after the disk name is changed. So we need to change the password file to take a new name with actual path. Use -f to force create the file otherwise the command will say already file exists as below
[oracle@xxxx ~]$ asmcmd pwcreate --asm '+DATA/orapwasm'
Enter password: ********
OPW-00010: Could not create the password file. A password file with this name is present.

ASMCMD-9454: could not create new password file
[oracle@xxxx ~]$ asmcmd pwcreate --asm '+DATA/orapwasm' -f
Enter password: ********
[oracle@xxxx ~]$ 
[oracle@xxxx ~]$ asmcmd pwget --asm
+DATA/orapwasm

You can now see that the password file is changed to the proper name. Still the connection fails as below.
[oracle@xxxx ~]$ sqlplus sys@abcdb:1522/+ASM as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 9 03:26:52 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Enter password:
ERROR:
ORA-01017: invalid username/password; logon denied

Now if we see the alert log, though the password file location is changed Oracle still looks for the old non-existent file. Snippet from alert log is as below
...
...
2020-06-09T03:30:14.571451-07:00
Errors in file /oracle/grid/diag/asm/+asm/+ASM/trace/+ASM_ora_23542.trc:
ORA-17503: ksfdopn:2 Failed to open file +ABC_DATA_DG01/orapwasm
ORA-15001: diskgroup "ABC_DATA_DG01" does not exist or is not mounted
ORA-06512: at line 4
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 679
ORA-06512: at line 2
ORA-01017: invalid username/password; logon denied
...
...
  
Solution:
In 19c, Oracle introduced a new clause for the ALTER SYSTEM command FLUSH PASSWORDFILE_METADATA_CACHE. So according to this clause if the location or the name of the password file changes, you must notify the database that a change has occurred. The command ALTER SYSTEM FLUSH PASSWORDFILE_METADATA_CACHE flushes the password file metadata cache stored in the SGA and informs the database that a change has occurred.
You may read more on all the ALTER SYSTEM commands in this link.

So once we do the flush, we are able to login to the database without any issues.
[oracle@xxxx ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 9 07:39:36 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0

SQL> alter system flush PASSWORDFILE_METADATA_CACHE;

System altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0
[oracle@xxxx ~]$ sqlplus sys@abcdb:1522/+ASM as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 9 07:40:46 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Enter password:
Last Successful login time: Tue Jun 09 2020 07:31:21 -07:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0

SQL>
 
Happy flushing..