Looking for something? Try here..

Tuesday, March 30, 2021

Ansible ping and types of Unix shells, a learning...

 I've just started to learn the basics of Ansible which is a powerful automation tool. The first step of any automation is to setup the environment on which we are working. I'm referring Frits Hoogland's blog which is very easy to understand the basics of Ansible. 

The first step is to set up your hosts file and test the connectivity between the servers. We have an inbuilt Ansible ping module to do this. Here is how my hosts file looks like 

[oracle@linux-8 ansible-project]$ cat hosts
192.168.56.131
192.168.56.151

[l7]
192.168.56.151

[l8]
192.168.56.131
192.168.56.181

[oracle@linux-8 ansible-project]$
I'm using the below command to test the ping and the result is also provided. Just keep in mind I have already setup password less communication and hence not required to pass the password. If I still need to prompt for password, I can provide -k option to the command line. 
[oracle@linux-8 ansible-project]$ ansible l8 -m ping -u oracle
SSH password:
[WARNING]: Unhandled error in Python interpreter discovery for host 192.168.56.131: unexpected output from Python interpreter discovery
192.168.56.131 | FAILED! => {
    "msg": "failed to transfer file to /home/oracle/.ansible/tmp/ansible-local-32681q1m7n_gx/tmpzvfmvwgz /home/oracle/.ansible/tmp/ansible-tmp-1617045011.6527164-32687-33868194395919/AnsiballZ_ping.py:\n\nOracle Home set as /u01/db/122 and SID=orcl\n"
}
192.168.56.181 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[oracle@linux-8 ansible-project]$
We can see the ping failed for 192.168.56.131 server with the error line including "\n\nOracle Home set as /u01/db/122 and SID=orcl\n"

This "\n\nOracle Home set as /u01/db/122 and SID=orcl\n" is a environment info message I have set in the server when we login as Oracle user using .bashrc file. 

Normal ssh works without any issues.
[oracle@linux-8 ansible-project]$ ssh 192.168.56.131
Web console: https://linux75.selvapc.com:9090/ or https://10.0.2.15:9090/

Last login: Mon Mar 29 14:46:24 2021 from 192.168.56.181
Oracle Home set as /u01/db/122 and SID=orcl
[oracle@linux75 ~]$
We can see the info message line once we login. 

All that Ansible ping command does is try to connect to host, verify a usable python and return pong on success. So, the question here is why my ssh command works without any issues but not my Ansible ping. 

The answer lies in how the connection is being established to the server. There are 4 types of shells in Unix. 
  • Interactive login                     - Login session started using graphical interface or by providing username/password via ssh
  • Interactive non-login             - Sessions started by opening another terminal in a server
  • Non interactive non-login     - Sessions started by scripts execution where the commands are run and exits the shell. We can't interact with these shells
  • Non interactive login     - Providing a command to run using ssh like "echo "some command" |ssh servername". Here instead of login, the session will run the "some command" and exits
So now, the difference in these shells is that the Login shells (shells started by the login system process, or by your X display manager) uses or reads .profile or .login files to set your environment or initialize your environment
Non interactive login/non-login shells, such as scripts started by shell or the Ansible ping that we are using now reads .bashrc or .kshrc files to initialize environment. Hence we should not set up any echo statements which would then become stdout to the shell initialization (which the shell won't expect) and errors out as a result. 

My .bashrc file looked like the below which was causing the issue with the non interactive session from Ansible ping. 
[oracle@linux75 ~]$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
export ORACLE_BASE=/u01/db
export ORACLE_HOME=/u01/db/122
export PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_SID=orcl
echo "Oracle Home set as $ORACLE_HOME and SID=$ORACLE_SID"

Once I remove the "echo" line from the file, I didn't have issues from the Ansible ping. We can simply move the echo line to the .bash_profile (I already had the entry in the .bash_profile file as well which is why when I previously did a ssh we got the info message once we login) which is the file used to initialize when we start interactive session. 
[oracle@linux-8 ansible-project]$ ansible l8 -m ping -u oracle -k
SSH password:
192.168.56.181 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
192.168.56.131 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[oracle@linux-8 ansible-project]$
You can notice now I have used -k option and the password is being prompted. Ping works as expected on all the servers in the group l8 and now we can proceed further with our automations..

Hope you have learnt regarding different types of Unix shells and the different initialization files they use. 

References: 

Happy Shelling!!! :)

Wednesday, March 3, 2021

Java HeadlessException on generating OSWatcher Analyzer graphs

 OS Watcher is a very handy tool in inspecting/diagnosing and troubleshooting any performance issues on the unix platforms. It is advised to run OSWatcher on every node that the Oracle instance is running and in the case of a performance issue, Oracle support can use this data to help diagnose performance problems which may outside the database. 


Installation and usage notes can be found with the following support notes
  • OSWatcher (Includes: [Video]) (Doc ID 301137.1)
  • OS Watcher User's Guide (Doc ID 1531223.1)
  • OSWatcher Analyzer User Guide (Doc ID 461053.1)
To outline, installation is very simple. One can download the latest version of OS Watcher Black Box or OSWbb (V8.4.0 at the time of writing this post) and transfer it to the server where the tool needs to be installed and untar the contents to the desired directory.

$ tar xvf oswbb.tar
$ chmod 744 *

The tool can be started by the below command

$ ./startOSWbb.sh 15 48 NONE /media/sf_Oracle/software/oswlinux-8
  • ARG1 = snapshot interval in seconds.
  • ARG2 = the number of hours of archive data to store.
  • ARG3 = (optional) the name of a compress utility to compress each file automatically after it is created.
  • ARG4 = (optional) an alternate (non default) location to store the archive directory.

OS Watcher Analyzer (oswbba) is java based utility that allows the user to graph and analyze data collected from running OSWatcher and it comes bundled with the oswbb tar file and no separate installation is needed. 
The latest version required java version 8 and higher. So make sure you have java with version 8 or higher is installed before running oswbba. 

oswbba can be started with the below command which prompts you for multiple options to display various graphs or to generate gifs or complete dashboard. 

$ java -jar oswbba.jar -i /location/of/archives 

This post is to cover the error encountered while starting the oswbba utility w.r.to java and how to overcome the same. 
[oracle@linux-8 oswbb]$ pwd
/oracle/osw/oswbb
[oracle@linux-8 oswbb]$ which java
/usr/bin/java
[oracle@linux-8 oswbb]$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
[oracle@linux-8 oswbb]$
I have already installed oswbb tool and I do have required java version installed as well.
While starting the tool and trying to generate the report, I encountered the below error. 
[oracle@linux-8 oswbb]$ java -jar oswbba.jar -i /media/sf_Oracle/software/oswlinux-8

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c)  2019 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...


Parsing file linux-8.selvapc.com_pidstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_pidstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_iostat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_iostat_21.03.02.0100.dat ...
This directory already exists. Rewriting...

Parsing file linux-8.selvapc.com_vmstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_vmstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_netstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_netstat_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_top_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_top_21.03.02.0100.dat ...

Parsing file linux-8.selvapc.com_ps_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_ps_21.03.02.0100.dat ...


Parsing Completed.


Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter 61 to Display Individual OS Process I/O RPS Graphs
Enter 62 to Display Individual OS Process I/O WPS Graphs
Enter 63 to Display Individual OS Process Percent User CPU Graphs
Enter 64 to Display Individual OS Process Percent System CPU Graphs
Enter 65 to Display Individual OS Process Percent Total CPU (User + System) Graphs
Enter 66 to Display Individual OS Process Percent Memory Graphs

Enter GP to Generate Individual Process Profile
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files

Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
Enter B to Returns to Baseline Graph Time Scale (Does not change analysis dataset)
Enter R to Remove Currently Displayed Graphs

Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:1
>>> Input Error: null
java.awt.HeadlessException
        at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
        at java.awt.Window.<init>(Window.java:536)
        at java.awt.Frame.<init>(Frame.java:420)
        at java.awt.Frame.<init>(Frame.java:385)
        at k.c(Unknown Source)
        at OSWGraph.OSWGraph.main(Unknown Source)
[oracle@linux-8 oswbb]$

There are 2 options to overcome this error. 

Option 1
If the tool is being run on a server with Oracle database software installed, we can use the java software that comes along with the database software. Make sure the java version is 8 or higher. 
* Output truncated for easy readability
[oracle@linux-8 oswbb]$ $ORACLE_HOME/jdk/bin/java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
[oracle@linux-8 oswbb]$
[oracle@linux-8 oswbb]$ $ORACLE_HOME/jdk/bin/java -jar oswbba.jar -i /media/sf_Oracle/software/oswlinux-8

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c)  2019 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...


Parsing file linux-8.selvapc.com_pidstat_21.03.02.0000.dat ...
Parsing file linux-8.selvapc.com_pidstat_21.03.02.0100.dat ...

...
...
...
Parsing file linux-8.selvapc.com_ps_21.03.02.0100.dat ...


Parsing Completed.


Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
...
...
...
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:1

Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
...
...
...
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:
Now, we don't see that error and the tool generated the graph and the output is displayed on the screen.
Sample graph below

Note: We need to have X11 forwarding enabled to get the output, else we will encounter the below error

Please Select an Option:1
>>> Input Error:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.

Option 2
If we are installing the oswbb tool on a server where there is no Oracle database software installed or if the bundled java along with the database software is of lower version, check the java package that is already installed on the server. 
By default, on RHEL 7 or RHEL 8, the OpenJDK software comes installed only with the headless package. 

[oracle@linux-8 oswbb]$ rpm -qa|grep java-1.8
java-1.8.0-openjdk-headless-1.8.0.282.b08-2.el8_3.x86_64

We can install java-1.8.0-openjdk package additional to the headless package and start running the oswbba tool to analyze the data collected to investigate performance issues. 

As I have run the tool on a server where Oracle software is installed, I used option 1 to overcome the error. 

Reference

Happy troubleshooting...!!!