There is no way we can delete the blackout via GUI as the option is grayed out. We have an option of stop but the result of stop is as below.
We can even try to stop or delete the black out via command lines but still we will get the below errors.
-sh-4.2$ -sh-4.2$ emcli get_blackout_details -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/" Status Status ID Run Jobs Next Start Duration Reason Frequency Repeat Days Months Start Time End Time TZ Region TZ Offset Type Full Blackout - Stop Failed 6 no none 02:00 Added via emctl utility once none none none 2020-08-16 10:05:06 2020-08-16 12:05:06 Etc/GMT +00:00 Blackout -sh-4.2$ -sh-4.2$ -sh-4.2$ emcli stop_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/" Error: Timed out while waiting for blackout to stop. Please check the blackout status from Enterprise Manager Cloud Control -sh-4.2$ -sh-4.2$ -sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/" Error: Cannot delete a blackout that has not ended or was not stopped -sh-4.2$
This is because, after the blackout has been started, we have stopped the agent and the server was put for decommission. Hence there is no agent to which the OMS can talk to perform stop or even delete the blackout created when the server/agent was alive.
Though this is not going to harm the EMCC operations in any way, the blackout will be listed as ACTIVE though we know that’s not the case. So better to end the blackout and cleanup the mess.
So how do we get out of this situation?
There are a few notes in Oracle support as below and a few more
EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)
These notes doesn’t apply to this scenario as we don’t have the target system for OMS to connect with.
We need to update the status manually for the blackouts which are hanging in the OMS. The work around provided in the support note Bug 30708506 : Remove historical blackouts in a scheduled status can be used to alter the status of the blackout to put it either to stopped (10) state or to ended (11) state after which the blackout can be deleted if needed.
The below operation has to be done on the OMS repository database.
-sh-4.2$ -sh-4.2$ sqlplus sysman/***** SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 2 09:33:45 2020 Version 19.8.0.0.0 Copyright (c) 1982, 2020, Oracle. All rights reserved. Last Successful login time: Mon Nov 02 2020 09:32:39 +00:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.8.0.0.0 SQL> SQL> SELECT UNIQUE target_name ,blackout_name ,STATUS ,start_time ,end_time ,target_type FROM ( SELECT a.blackout_name ,a.start_time ,a.end_time ,a.STATUS ,a.target_name ,a.target_type FROM sysman.MGMT$BLACKOUT_HISTORY a ) WHERE STATUS = 'Stop Failed' ORDER BY start_time; 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 no rows selected SQL> select blackout_guid from em_blackouts_ng where blackout_name='CMQ_2020-08-16 10:05:06'; BLACKOUT_GUID -------------------------------- 1A0C988C8BC4932A6D14F2B36E63D472 SQL> UPDATE em_blackout_target_state SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472'; 1 row updated. SQL> UPDATE em_blackouts_ng SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472'; 1 row updated. SQL> commit; Commit complete. SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.8.0.0.0 -sh-4.2$You can now see the status is updated as “Ended” in EMCC as well.
An attempt to delete the blackout now gets completed successfully
-sh-4.2$ -sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://*****:3872/emd/main/" Blackout "CMQ_2020-08-16 10:05:06" deleted successfully -sh-4.2$
References:
EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1) EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)
Bug 30708506 : Remove historical blackouts in a scheduled status
https://docs.oracle.com/cd/E63000_01/EMVWS/blackoutviews.htm#EMVWS32043
https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emcli/em-cli-verbs.html
Happy Work arounding!!!
This was exactly an issue we had. Thank you for digging through all the Support DOC ID's. I was getting lost there!
ReplyDeleteI'm glad it helped you! :)
DeleteHad the same issue and followed your steps! Success, the blackout was ended and removed successfully. Thank you for publishing these steps.
ReplyDeleteI'm glad it helped you!!
Delete