Wednesday, January 20, 2021

DNS server using BIND9 for Home Lab

Happy New Year to all my readers. This is my first post of this year and I've started my year by building a RAC environment for the lab. 

Most of us who wants to learn building Oracle RAC and its operations use Oracle VirtualBox as the virtualization software as it is free to use. You can also take a look at Workstation Pro which is a very powerful virtualization product (requires license and 60 day trail available) from VMware which is used in many companies as well or their free to use software Workstation Player which is similar to Workstation Pro with a few limitations. 

One main requirement of building the RAC environment (For Oracle database versions 11gR2 or higher) is the availability of a DNS server or Oracle GNS which is used to resolve SCAN name minimum 1 up to 3 or more IP addresses (generally an odd number due high availability requirements) in a round robin fashion. 

Most of the beginners use Windows OS to work on the practices until they get familiar with other OS such as Unix/Linux and hence we in this post look at how can we configure the host Windows OS to server as the DNS server for the Guest OS running on the virtualization software.

Software Installation: 

We will use BIND 9 to create our DNS server on the Windows host. Download the latest version of BIND 9 from the download page and then extract the contents into a folder. I'm using 7zip software to extract and you can use any software as you wish


Once the contents are extracted, go the extracted files folder and start BINDInstall.exe as Run as Administrator


Set the Target Directory, provide service account password and confirm and click on Install. 


The Target Directory provided is the software location or the BIND9 path. Once the installation is complete, click OK and exit the installer. 

Configuration: 

The software is installed just a few clicks. Now let us look into the configuration part. In this post we will configure a very basic DNS server config and you may have to look into the documentation of the particular software version that you have downloaded if you like to configure a sophisticated setup. 

Once the software is installed, you will be able to see bin directory and etc directory under the software installation path, C:\Program Files\BIND9 in my case. We will have to create the named.conf file under etc directory. named.conf is the name server configuration file containing collection of statements using nested options surrounded by opening and closing ellipse characters, { }. 

Sample reference file of my host's named.conf configuration looks like the below

// Listen only on this machine i.e localhost or from 192.168.56.1
options {
  listen-on port 53 { 127.0.0.1; 192.168.56.1; }; 
  directory "C:\Program Files\BIND9\zones";
  allow-transfer { none; };
  recursion no;
  forwarders { 194.168.43.1; };
};

zone "selvapc.com." IN {
  type master;
  file "selvapc.com.zone";
  allow-transfer { none; };
};

zone "56.168.192.in-addr.arpa." IN {
  type master;
  file "56.168.192.in-addr.arpa";
  allow-update { none; };
};
Explanation of the above: 
Line 1: Starts with // is a comment line
Line 3: DNS server is listening only on port 53 and from local host or 192.168.56.1. This ip is the default VirtualBox Host only adapter ip address which is used by the guest OS to communicate with Windows host. You can also mention 192.168.0.0/24 to listen on all ranges of ip in 192.168

Line 4: It is the named working directory
Line 5: Specifies which hosts are allowed to receive zone transfers from the server
Line 6: Prevents new data from being cached as an effect of client queries
Line 7: Specifies a list of IP addresses to which queries are forwarded. We have only limited our DNS to have a specific IP address used for SCAN. If any other name is queried, it will be forwarded to this set of IPs specified. I have provided the host machines DNS server IP provided by ISP which will communicate with internet to resolve names. You can obtain this by checking ipconfig /all in command prompt. Note: This is not the public DNS address from my ISP. You can also provide cloudfare's (1.1.1.1) or Google's (8.8.8.8) DNS as well to query internet if required.

Line 10: Zone statement for selvapc.com zone. If you don't have a domain name defined, you can use localdomain instead of selvapc.com
Line 11 to 13: Zone statement type is master and is instructed to read selvapc.com.zone file which will be present in zones directory under software directory path. Other hosts can't update this file
Line 16: This file will be used to do the reverse lookup. This is optional as RAC will need only the forward lookup to resolve the IP address defined under SCAN name. DNS queries issued under 192.168.56.* will be reverse resolved
Line 17 to 19: type is master, instructed to read 56.168.192.in-addr.arpa in zones directory under software directory path. Other hosts can't update this file

Now as we have defined the named.conf file, we need to create the zones file. Create a directory by name zones under software directory path, C:\Program Files\BIND9\zones in my case. 

The 2 files will look like below. 
$TTL    86400
@               IN SOA  localhost root.localhost (
                                        42              ; serial (d. adams)
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum
                IN NS           localhost
localhost       	 IN A            127.0.0.1
12r1-rac1            IN A    192.168.56.101
12r1-rac2            IN A    192.168.56.102
12r1-rac1-priv       IN A    192.168.1.101
12r1-rac2-priv       IN A    192.168.1.102
12r1-rac1-vip        IN A    192.168.56.103
12r1-rac2-vip        IN A    192.168.56.104
scan-12r1        	 IN A    192.168.56.105
scan-12r1        	 IN A    192.168.56.106
scan-12r1        	 IN A    192.168.56.107
You can see here all these IPs can be resolved to their respective names. SCAN name has 3 IPs configured which will resolve in round robin fashion. 
$ORIGIN 56.168.192.in-addr.arpa.
$TTL 1H
@       IN      SOA     selvapc.com.     root.selvapc.com. (      2
                                                3H
                                                1H
                                                1W
                                                1H )
56.168.192.in-addr.arpa.         IN NS      selvapc.com.

101     IN PTR  12r1-rac1.selvapc.com.
102     IN PTR  12r1-rac2.selvapc.com.
103     IN PTR  12r1-rac1-vip.selvapc.com.
104     IN PTR  12r1-rac2-vip.selvapc.com.
105     IN PTR  scan-12r1.selvapc.com.
106     IN PTR  scan-12r1.selvapc.com.
107     IN PTR  scan-12r1.selvapc.com.
Note, the trailing '.' is very important at all the places, else the configuration won't work as expected.

Now the configuration part is completed. We have 1 more step remaining which is adjusting the firewall rule to allow TCP and UDP to port 53. This can be done by following the step below. 
  • Go to Control Panel >> System and Security >> Windows Defender Firewall >> Click Advanced Settings
  • Click Inbound Rules >> New Rule >> Protocol and Ports (Port) >> Next >> TCP >> Specific local ports (53) >> Next >> Allow the connection >> Next >> Check all the check boxes >> Next >> Specify a name for the rule Eg: TCP53 >> Finish
  • Click Inbound Rules >> New Rule >> Protocol and Ports (Port) >> Next >> UDP >> Specific local ports (53) >> Next >> Allow the connection >> Next >> Check all the check boxes >> Next >> Specify a name for the rule Eg: UDP53 >> Finish
We are now on to our final step of the configuration is start or restart of the ISC BIND service under windows. 
  • Windows+R >> Type "services.msc" >> Click OK
  • Search for ISC BIND service. Click Start if it is not running or Restart if it is already running

We have successfully completed the set up of DNS server on our Windows host machine. 

Verification: 

We now have to verify if the setup works as expected. Issue the nslookup command as below. 

We can now see that the SCAN name is able to resolve to 3 IP addresses. As we want this name resolution to happen in the local host or 192.168.56.1 (which serves as the IP address for the host OS in VirtualBox), we pass the third argument as localhost or 192.168.56.1. 

If you install Linux guest OS, the /etc/resolv.conf file should contain 192.168.56.1 as nameserver defined which will then resolve the SCAN name without any issues. Here is an example content and o/p from my lab environment. 
[oracle@12r1-rac1 ~]$
[oracle@12r1-rac1 ~]$ uname -a
Linux 12r1-rac1.selvapc.com 3.8.13-98.4.1.el6uek.x86_64 #2 SMP Wed Sep 23 18:46:01 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[oracle@12r1-rac1 ~]$ cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain selvapc.com
nameserver 192.168.56.1
[oracle@12r1-rac1 ~]$ nslookup scan-12r1
Server:         192.168.56.1
Address:        192.168.56.1#53

Name:   scan-12r1.selvapc.com
Address: 192.168.56.107
Name:   scan-12r1.selvapc.com
Address: 192.168.56.106
Name:   scan-12r1.selvapc.com
Address: 192.168.56.105

[oracle@12r1-rac1 ~]$ nslookup 192.168.56.106
Server:         192.168.56.1
Address:        192.168.56.1#53

106.56.168.192.in-addr.arpa     name = 12r1-scan.selvapc.com.

[oracle@12r1-rac1 ~]$

All is good now and we can start building our Oracle RAC databases.. 

References: 
Happy BINDing...!!! 

Wednesday, November 4, 2020

Enterprise Manager Cloud Control - Blackout - manual update in OMS

I have created the blackout for one of the servers with duration of 2 hours. Even after a month, still the server pops up in the Blackouts window of the EMCC. It is clearly seen the status is in “Stop Failed” 



There is no way we can delete the blackout via GUI as the option is grayed out. We have an option of stop but the result of stop is as below. 



We can even try to stop or delete the black out via command lines but still we will get the below errors. 
-sh-4.2$ 
-sh-4.2$ emcli get_blackout_details -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Status                       Status ID  Run Jobs  Next Start  Duration  Reason                   Frequency  Repeat  Days  Months  Start Time           End Time             TZ Region  TZ Offset  Type
Full Blackout - Stop Failed  6          no        none        02:00     Added via emctl utility  once       none    none  none    2020-08-16 10:05:06  2020-08-16 12:05:06  Etc/GMT    +00:00     Blackout
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli stop_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Timed out while waiting for blackout to stop. Please check the blackout status from Enterprise Manager Cloud Control
-sh-4.2$
-sh-4.2$
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://xxxxx:3872/emd/main/"
Error: Cannot delete a blackout that has not ended or was not stopped
-sh-4.2$

This is because, after the blackout has been started, we have stopped the agent and the server was put for decommission. Hence there is no agent to which the OMS can talk to perform stop or even delete the blackout created when the server/agent was alive.

Though this is not going to harm the EMCC operations in any way, the blackout will be listed as ACTIVE though we know that’s not the case. So better to end the blackout and cleanup the mess.

So how do we get out of this situation?

There are a few notes in Oracle support as below and a few more

EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)

These notes doesn’t apply to this scenario as we don’t have the target system for OMS to connect with.

We need to update the status manually for the blackouts which are hanging in the OMS. The work around provided in the support note Bug 30708506 : Remove historical blackouts in a scheduled status can be used to alter the status of the blackout to put it either to stopped (10) state or to ended (11) state after which the blackout can be deleted if needed.

The below operation has to be done on the OMS repository database. 
-sh-4.2$ 
-sh-4.2$ sqlplus sysman/*****

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 2 09:33:45 2020
Version 19.8.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Last Successful login time: Mon Nov 02 2020 09:32:39 +00:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL>
SQL> SELECT UNIQUE target_name
        ,blackout_name
        ,STATUS
        ,start_time
        ,end_time
        ,target_type
FROM (
        SELECT a.blackout_name
                ,a.start_time
                ,a.end_time
                ,a.STATUS
                ,a.target_name
                ,a.target_type
        FROM sysman.MGMT$BLACKOUT_HISTORY a
        )
WHERE STATUS = 'Stop Failed'
ORDER BY start_time;  2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17

no rows selected

SQL> select blackout_guid from em_blackouts_ng where blackout_name='CMQ_2020-08-16 10:05:06';

BLACKOUT_GUID
--------------------------------
1A0C988C8BC4932A6D14F2B36E63D472

SQL> UPDATE em_blackout_target_state SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> UPDATE em_blackouts_ng SET blackout_status = 11 WHERE blackout_guid = '1A0C988C8BC4932A6D14F2B36E63D472';

1 row updated.

SQL> commit;

Commit complete.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
-sh-4.2$ 
You can now see the status is updated as “Ended” in EMCC as well.

An attempt to delete the blackout now gets completed successfully
-sh-4.2$ 
-sh-4.2$ emcli delete_blackout -name="CMQ_2020-08-16 10:05:06" -createdby="<SYSTEM>" -emd_url="https://*****:3872/emd/main/"
Blackout "CMQ_2020-08-16 10:05:06" deleted successfully
-sh-4.2$

The clean up is now done and we can peacefully sleep :)

References: 
EM 12c, 13c : Blackout Stuck in 'Stop Pending' Status (Doc ID 2014040.1)
EM 13c: Enterprise Manager 13c Cloud Control HAS Targets Show: Diagnose For Status Pending-Post Blackout (Doc ID 2528161.1)
Cannot Stop Emctl Blackouts (Started From Emctl) (Doc ID 786933.1)
Bug 30708506 : Remove historical blackouts in a scheduled status
https://docs.oracle.com/cd/E63000_01/EMVWS/blackoutviews.htm#EMVWS32043
https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emcli/em-cli-verbs.html

Happy Work arounding!!!