Enterprise Manager Cloud Control - Blackout - manual update in OMS

I have created the blackout for one of the servers with duration of 2 hours. Even after a month, still the server pops up in the Blackouts window of the EMCC. It is clearly seen the status is in “Stop Failed” 

There is no way we can delete the blackout via GUI as the option is grayed out. We have an option of stop but the result of stop is as below. 

We can even try to stop or delete the black out via command lines but still we will get the below errors. 

-sh-4.2$ 
-sh-4.2$ emcli get_blackout_details -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Status                       Status ID  Run Jobs  Next Start  Duration  Reason                   Frequency  Repeat  Days  Months  Start Time           End Time             TZ Region  TZ Offset  Type
Full Blackout - Stop Failed  6          no        none        02:00     Added via emctl utility  once       none    none  none    2020-08-16 10:05:06  2020-08-16 12:05:06  Etc/GMT    +00:00     Blackout
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli stop_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Timed out while waiting for blackout to stop. Please check the blackout status from Enterprise Manager Cloud Control
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Cannot delete a blackout that has not ended or was not stopped
-sh-4.2$

This is because, after the blackout has been started, we have stopped the agent and the server was put for decommission. Hence there is no agent to which the OMS can talk to perform stop or even delete the blackout created when the server/agent was alive.

Though this is not going to harm the EMCC operations in any way, the blackout will be listed as ACTIVE though we know that’s not the case. So better to end the blackout and cleanup the mess.

So how do we get out of this situation?

There are a few notes in Oracle support as below and a few more

EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)

These notes doesn’t apply to this scenario as we don’t have the target system for OMS to connect with.

We need to update the status manually for the blackouts which are hanging in the OMS. The work around provided in the support note Bug 30708506 : Remove historical blackouts in a scheduled status can be used to alter the status of the blackout to put it either to stopped (10) state or to ended (11) state after which the blackout can be deleted if needed.

The below operation has to be done on the OMS repository database. 

-sh-4.2$ 
-sh-4.2$ sqlplus sysman/*****

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 2 09:33:45 2020
Version 19.8.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Last Successful login time: Mon Nov 02 2020 09:32:39 +00:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL>
SQL> SELECT UNIQUE target_name
        ,blackout_name
        ,STATUS
        ,start_time
        ,end_time
        ,target_type
FROM (
        SELECT a.blackout_name
                ,a.start_time
                ,a.end_time
                ,a.STATUS
                ,a.target_name
                ,a.target_type
        FROM sysman.MGMT$BLACKOUT_HISTORY a
        )
WHERE STATUS = 'Stop Failed'
ORDER BY start_time;  2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17

no rows selected

SQL> select blackout_guid from em_blackouts_ng where blackout_name='CMQ_2020-08-16 10:05:06';

BLACKOUT_GUID
--------------------------------
1A0C988C8BC4932A6D14F2B36E63D472

SQL> UPDATE em_blackout_target_state SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> UPDATE em_blackouts_ng SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> commit;

Commit complete.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
-sh-4.2$ 

You can now see the status is updated as “Ended” in EMCC as well.

An attempt to delete the blackout now gets completed successfully

-sh-4.2$ 
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://*****:3872/emd/main/"
Blackout "CMQ_2020-08-16 10:05:06" deleted successfully
-sh-4.2$

The clean up is now done and we can peacefully sleep :)

References: 

EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)
Bug 30708506 : Remove historical blackouts in a scheduled status

https://docs.oracle.com/cd/E63000_01/EMVWS/blackoutviews.htm#EMVWS32043

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emcli/em-cli-verbs.html

Happy Work arounding!!!