Saturday, May 1, 2021

Broken 18c GI upgrade with ORA-29702 and fixes

 Ok, I was very confident the upgrade would succeed when the upgrade started to run after fixing all the necessary pre requisites and is running without issues. I also took a screen shot of the progress as below after starting the rootupgrade.sh See software version already in 18.0.0.0.0


Then after few minutes of my wait, the upgrade failed on node 1. When the rootupgrade.sh script was started and running during Executing upgrade step 17 of 19: 'UpgradeCluster' step, the below error was encountered. 

CRS-2672: Attempting to start 'ora.asm' on '12r1-rac1'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4150
Additional information: 67028943
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/oradir1/grid/base/diag/crs/12r1-rac1/crs/trace/ohasd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.asm' on '12r1-rac1' failed
CRS-2679: Attempting to clean 'ora.asm' on '12r1-rac1'
CRS-2681: Clean of 'ora.asm' on '12r1-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on '12r1-rac1'
CRS-2883: Resource 'ora.storage' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2021/04/25 17:38:06 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /oradir1/grid/18/crs/install/crsupgrade.pm line 1462.

The alert log of the +ASM1 of 18c home has the following errors. 

MMNL started with pid=29, OS id=26058
2021-04-25T17:27:01.603535+05:30
Unable to start the instance due to software incompatiblity
2021-04-25T17:27:01.604426+05:30
ORA-15156: cluster in rolling upgrade from version [12.1.0.1.0] to [18.0.0.0.0]

2021-04-25T17:27:01.853249+05:30
Error: Shutdown in progress. Error: 29702.
USER (ospid: 25746): terminating the instance due to ORA error 29702
2021-04-25T17:27:02.925386+05:30
Instance terminated by USER, pid = 25746

Detailed info is available in /oradir1/grid/base/diag/crs/12r1-rac1/crs/trace/ohasd_oraagent_oracle.trc file which has reported the below errors.

...
...
...
...
2021-04-25 17:27:14.156 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection:connectInt 200 connected
2021-04-25 17:27:14.156 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 040 resname:ora.asm startup
2021-04-25 17:27:17.534 :CLSFRAME:2796190720:  TM [MultiThread] is changing desired thread # to 5. Current # is 4
2021-04-25 17:27:17.534 :CLSDYNAM:2539628288: [ora.evmd]{0:0:2} [check] DaemonAgent::check returned 0
2021-04-25 17:27:17.534 :CLSDYNAM:2539628288: [ora.evmd]{0:0:2} [check] Deep check returned 0
2021-04-25 17:27:22.679 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] ORA-29702: error occurred in Cluster Group Service operation

2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 050 errcode 29702
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 160 ORA-29701 or ORA-29702 instance dumping m_instanceType:2 m_lastOCIError:29702
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 170 ORA-29701 or ORA-29702 shutdown abort
2021-04-25 17:27:22.680 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::shutdown mode 4
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 260 disConnect
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 020 connect logmode:8008
2021-04-25 17:27:32.681 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::connectInt 020 server not attached
2021-04-25 17:27:33.726 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection:connectInt 200 connected
2021-04-25 17:27:33.727 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 040 resname:ora.asm startup
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] ORA-29702: error occurred in Cluster Group Service operation

2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 050 errcode 29702
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 160 ORA-29701 or ORA-29702 instance dumping m_instanceType:2 m_lastOCIError:29702
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstAgent::startInstance 170 ORA-29701 or ORA-29702 shutdown abort
2021-04-25 17:27:42.263 :CLSDYNAM:2754123520: [ ora.asm]{0:5:3} [start] InstConnection::shutdown mode 4
...
...
...

This is a weird error right. Error says unable to start the instance due to software incompatibility and you go make a search on Oracle support for "ORA 29702 during GI upgrade", it returns n number of issues with references to a few bugs as well. Nothing seems to be fitting the issue that I have encountered. 
One more interesting thing is that, if I set the environment variables to 18c home and start the ASM instance, the instance starts without issue. So I then again shutdown the ASM instance and restarted the rootupgrade.sh script. This time too the same error occurs when Oracle tries to start ASM and the upgrade fails. So, after long search and different attempts, I was not able to fix the issue. I cancelled the upgrade session and decided to look into the issue next day. 

On the next day when I start both the nodes, I see that the ASM instance on the first node is up from 18c home and no services started on the second node. I decided to continue the upgrade from where I left. Since we already completed the software set up and our error occurred during the execution of rootupgrade.sh script, I started from the same step after shutting down the entire cluster running on node 1. 
This time the rootupgrade.sh completed on node 1. Shut down and startup of the entire server did the trick it seems. 

So I continued with the upgrade by running the rootupgrade.sh on node 2 which also succeeded without any issues. Finally ran the below command to complete the upgrade. 

$GRID_HOME/gridSetup.sh -executeConfigTools -all -silent -responseFile responsefile.rsp

Well, now after everything has been done, the command ended with the below error. 

You can find the logs of this session at:
/u01/oraInventory/logs/GridSetupActions2021-04-29_01-25-13AM

Configuration failed.
[WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped.
   ACTION: Refer to the logs or contact Oracle Support Services.

$ 

We can also see the below response when querying the active version

[oracle@12r1-rac1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
[oracle@12r1-rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [UPGRADE FINAL]. The cluster active patch level is [70732493].
[oracle@12r1-rac1 ~]$

This is a new change in GI starting 18c where cluvfy performs a post upgrade check and reports the status and doesn't change the status from [UPGRADE FINAL] to [NORMAL] if any of the post upgrade check fails. 

In my case, opening the detailed log /u01/oraInventory/logs/GridSetupActions2021-04-29_01-25-13AM/gridSetupActions2021-04-29_01-25-13AM.log shows the below errors (trimmed output)

...
...
...
INFO:  [Apr 29, 2021 1:42:07 AM] Skipping line: Verifying Single Client Access Name (SCAN) ...PASSED
INFO:  [Apr 29, 2021 1:42:11 AM] Verifying OLR Integrity ...PASSED
INFO:  [Apr 29, 2021 1:42:11 AM] Skipping line: Verifying OLR Integrity ...PASSED
INFO:  [Apr 29, 2021 1:42:17 AM] Verifying Voting Disk ...FAILED (PRVF-5431, PRVF-5449, PRVG-2032)
INFO:  [Apr 29, 2021 1:42:17 AM] Skipping line: Verifying Voting Disk ...FAILED (PRVF-5431, PRVF-5449, PRVG-2032)
INFO:  [Apr 29, 2021 1:42:17 AM] Verifying ASM Integrity ...
INFO:  [Apr 29, 2021 1:42:17 AM] Skipping line: Verifying ASM Integrity ...
INFO:  [Apr 29, 2021 1:42:17 AM]   Verifying Node Connectivity ...
...
...
...
INFO:  [Apr 29, 2021 1:42:22 AM] Skipping line: Verifying Device Checks for ASM ...
INFO:  [Apr 29, 2021 1:42:23 AM]   Verifying Access Control List check ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:23 AM] Skipping line:   Verifying Access Control List check ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:28 AM]   Verifying I/O scheduler ...PASSED
INFO:  [Apr 29, 2021 1:42:28 AM] Skipping line:   Verifying I/O scheduler ...PASSED
INFO:  [Apr 29, 2021 1:42:28 AM] Verifying Device Checks for ASM ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:28 AM] Skipping line: Verifying Device Checks for ASM ...FAILED (PRVF-9992)
INFO:  [Apr 29, 2021 1:42:30 AM] Verifying ASM disk group free space ...PASSED
INFO:  [Apr 29, 2021 1:42:30 AM] Skipping line: Verifying ASM disk group free space ...PASSED
INFO:  [Apr 29, 2021 1:42:30 AM] Verifying User Not In Group "root": oracle ...PASSED
...
...
...

According to support note Doc ID 2583141.1, fix the critical issues that cluvfy post upgrade check reports and rerun cluvfy with -collect cluster to update the status to [NORMAL]. 

[oracle@12r1-rac1 ~]$ $ORACLE_HOME/bin/cluvfy stage -post crsinst -gi_upgrade -n all

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.56.0" ...PASSED
Verifying Node Connectivity ...PASSED
...
...
...
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...FAILED (PRVF-9992)
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...FAILED (PRVF-9992)
...
...
...
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following nodes:
        12r1-rac2,12r1-rac1


Failures were encountered during execution of CVU verification request "stage -post crsinst".

Verifying Voting Disk ...FAILED
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed

12r1-rac2: PRVF-5449 : Check of Voting Disk location "/dev/oracleasm/disk1"
           failed on the following nodes:
12r1-rac2: PRVG-2032 : Group of file "/dev/oracleasm/disk1" did not match the
           expected value on node "12r1-rac2". [Expected = "oinstall" ; Found =
           "dba"]

12r1-rac1: PRVF-5449 : Check of Voting Disk location "/dev/oracleasm/disk1"
           failed on the following nodes:
12r1-rac1: PRVG-2032 : Group of file "/dev/oracleasm/disk1" did not match the
           expected value on node "12r1-rac1". [Expected = "oinstall" ; Found =
           "dba"]

Verifying Device Checks for ASM ...FAILED
  Verifying Access Control List check ...FAILED
  12r1-rac2: PRVF-9992 : Group of device "/dev/oracleasm/disk5" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
             nodes: [12r1-rac2, 12r1-rac1]
  12r1-rac2: PRVF-9992 : Group of device "/dev/oracleasm/disk4" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
...
...
...
  12r1-rac1: PRVF-9992 : Group of device "/dev/oracleasm/disk8" did not match
             the expected group. [Expected = "oinstall"; Found = "dba"] on
             nodes: [12r1-rac2, 12r1-rac1]


CVU operation performed:      stage -post crsinst
Date:                         May 1, 2021 7:37:41 PM
CVU home:                     /oradir1/grid/18/
User:                         oracle
[oracle@12r1-rac1 ~]$

We can see most of the errors are due to group expectation on the ASM disks. I just changed the disk ownership to oracle:oinstall instead of dba by editing the udev rules and reloading and reran the cluvfy for post upgrade which succeeded without issues. 
Next step is to run cluvfy with -collect cluster to update the status of the GI upgrade (trimmed output)

[oracle@12r1-rac1 ~]$ $ORACLE_HOME/bin/cluvfy stage -post crsinst -collect cluster -gi_upgrade -n all

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.56.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
...
...
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED

Collecting OS mandatory requirements baseline

Collecting Physical Memory ...collected
Collecting Available Physical Memory ...collected
...
...
...
Collecting Package: ksh ...collected
Collecting Package: make-3.81 ...collected
...
...
...
Collecting HugePages Existence ...collected
Collecting Hardware Clock synchronization at shutdown ...collected
Collecting availability of port 8888 ...collected
Collecting Ethernet Jumbo Frames ...collected
...
...
...
Collecting CSS reboottime parameter ...collected
Collecting CSS disktimeout parameter ...collected


Verifying ASM mandatory requirements

Collecting ASM Initialization Parameters ...collected
Collecting ASM Configuration Collection ...collected
Collecting ASM Disk Group Configuration Collection ...collected


Verifying ASM best practices

Collecting ASM_POWER_LIMIT ...collected
...
...
...
Collecting ASM disk rebalance operations in WAIT status ...collected

Baseline collected.
Collection report for this execution is saved in file "/oradir1/grid/base/crsdata/@global/cvu/baseline/install/grid_install_18.0.0.0.0.zip".

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         May 1, 2021 8:22:45 PM
CVU home:                     /oradir1/grid/18/
User:                         oracle
[oracle@12r1-rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [70732493].
[oracle@12r1-rac1 ~]$

We can see now the cluster upgrade state is [NORMAL] meaning the GI upgrade is a success!

Foot Note: 
This blog post is the one which is closest match to the error which we encountered where the author says regarding Bug 21484367 but I doubt this might have matched as I have not configured so many services as this system being my home lab. Also ASM instance came up without issues after starting the node the next day (the author says after he killed ohasd and oraagent.bin process the GI and evmd came up). I didn't get a chance to try this anyhow. 

References: 
  • How to resolve the cluster upgrade state of [UPGRADE FINAL] after successfully upgrading Grid Infrastructure (GI) to 18c or higher (Doc ID 2583141.1)
  • How to Install / Upgrade/ Clone 12.2 Grid Infrastructure in Silent Mode Using gridSetup.sh (Doc ID 2327772.1)
  • evmd not starting in oracle restart (pfierens.blogspot.com)
  • Various Oracle support notes popping up with search term "ORA 29702 during GI upgrade"

Happy upgrading...!!!

Monday, April 19, 2021

LVM on Linux demo to add disk and create mount points

 Logical Volume Management (LVM) is very powerful device mapper framework that provides logical volume management for Linux systems. 

In this post I'll just present you the minimal commands that are required to make use of the storage that is added to OS. In my last post, I've written about how to add storage to guest OS running on a VirtualBox. It can be compared to the storage addition done at OS level in companies. Once the storage is provided to the OS (be it a single disk added or a chunk from SAN, etc.,), we will look at how to make the storage available at the Linux OS for usage. 


The process will involve the below steps

  • Create Linux LVM partition using fdisk
  • Create physical volume
  • Create Volume group
  • Create Logical Volume
  • Create file system (ext4 used in this example)
  • Mount the FS
  • Make the FS mount permanent

Explanation of commands (highlighted lines) used are provided as comment lines prior to the commands in the demo. A few notable result lines are also highlighted. 

 
[root@12r1-rac2 ~]# #check for available File systems
[root@12r1-rac2 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  6.0G   18G  26% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  267G  234G  54% /media/sf_Oracle
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #List disks available, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

...
...
...
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00056a91

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3264    26214400   83  Linux
/dev/sda2            3264        4830    12579840   83  Linux
/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Format the newly added disk. 
[root@12r1-rac2 ~]# #I'm creating 2 primary partitions with 15GB each in total of 30GB disk
[root@12r1-rac2 ~]# #Using n,p,1,1,+15G,n,p,2(default),1960(default),3916(default),w as input
[root@12r1-rac2 ~]# #This step can be skipped if you want to partition while initializing using pvcreate
[root@12r1-rac2 /]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3916, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-3916, default 3916): +15G

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1960-3916, default 1960):
Using default value 1960
Last cylinder, +cylinders or +size{K,M,G} (1960-3916, default 3916):
Using default value 3916

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 /]#
[root@12r1-rac2 ~]# #List the available disks. Newly created disks are visible, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

/dev/sda3            4830        6136    10485760   83  Linux
/dev/sda4            6136        6528     3147776    5  Extended
/dev/sda5            6136        6528     3145728   82  Linux swap / Solaris

Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   83  Linux
/dev/sdf2            1960        3916    15719602+  83  Linux
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Change the partition type of /dev/sdf1, /dev/sdf2 to Linux LVM 
[root@12r1-rac2 ~]# #Using t,1,8e,t,2,8e,w as input
[root@12r1-rac2 ~]# fdisk /dev/sdf

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the available disks, o/p truncated for better viewing
[root@12r1-rac2 ~]# fdisk -l

Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5f998c79

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         652     5237158+  83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes
...
...
...

Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf306e413

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         652     5237158+  83  Linux

Disk /dev/sdf: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9e53da19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1        1959    15735636   8e  Linux LVM
/dev/sdf2            1960        3916    15719602+  8e  Linux LVM
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Currently LVM is not configured in the system and hence no o/p for the below 3 commands
[root@12r1-rac2 ~]# pvs
[root@12r1-rac2 ~]# vgs
[root@12r1-rac2 ~]# lvs
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#	#Create physical volumes using available disks. (also used to format block device if not formatted already)
[root@12r1-rac2 ~]# pvcreate /dev/sdf1 /dev/sdf2
  Physical volume "/dev/sdf1" successfully created
  Physical volume "/dev/sdf2" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the physical volumes, you can use 'pvdisplay' command to view detailed information
[root@12r1-rac2 ~]# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sdf1       lvm2 ---  15.01g 15.01g
  /dev/sdf2       lvm2 ---  14.99g 14.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Create Volume Group (named vg18c) using the available PVs 
[root@12r1-rac2 ~]# vgcreate vg18c /dev/sdf1 /dev/sdf2
  Volume group "vg18c" successfully created
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List available VGs, #PV denotes 2 PVs associated with this VG. 
[root@12r1-rac2 ~]# #Use 'vgdisplay' command to view detailed information
[root@12r1-rac2 ~]# vgs
  VG    #PV #LV #SN Attr   VSize  VFree
  vg18c   2   0   0 wz--n- 29.99g 29.99g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #Create Logical Volume from the available VG
[root@12r1-rac2 ~]# # -L is size and -n is name. Last variable is the VG, the LV will be created from.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cgrid vg18c
  Logical volume "18cgrid" created.
[root@12r1-rac2 ~]# lvcreate -L 12G -n 18cdb vg18c
  Logical volume "18cdb" created.
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# #List the LVs. Use 'lvdisplay' to view detailed information
[root@12r1-rac2 ~]# lvs
  LV      VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  18cdb   vg18c -wi-a----- 12.00g
  18cgrid vg18c -wi-a----- 12.00g
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Now create File system of type ext4 with the available LV using mkfs command
[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cdb
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]# mkfs -t ext4 /dev/vg18c/18cgrid
mke2fs 1.43-WIP (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
786432 inodes, 3145728 blocks
157286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=3221225472
96 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# mkdir /oradir1 /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Mount the FS using mount command
[root@12r1-rac2 ~]# mount /dev/vg18c/18cgrid /oradir1
[root@12r1-rac2 ~]# mount /dev/vg18c/18cdb /oradir2
[root@12r1-rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  6.0G   18G  26% /
tmpfs                 2.0G  629M  1.4G  32% /dev/shm
/dev/sda3             9.8G  5.9G  3.4G  64% /u01
/dev/sda2              12G  4.9G  6.3G  44% /u02
Oracle                500G  267G  234G  54% /media/sf_Oracle
/dev/mapper/vg18c-18cgrid
                       12G   30M   12G   1% /oradir1
/dev/mapper/vg18c-18cdb
                       12G   30M   12G   1% /oradir2
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# #Edit the fstab to make the mount permanent across reboots. 
[root@12r1-rac2 ~]# vi /etc/fstab
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]#
[root@12r1-rac2 ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Oct 20 01:50:26 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b149969c-329d-48d7-ace0-ea7ac6671964 /                       ext4    defaults        1 1
UUID=aa9b0bed-4eab-407d-8c0f-c2739a7678fb /u01                    ext4    defaults        1 2
UUID=2ab60579-5059-4c1e-9db1-7ac8d9a5ac77 /u02                    ext4    defaults        1 2
UUID=6ebf30ec-79d9-4c54-8e44-ce7f47190fdd swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/mapper/vg18c-18cgrid     /oradir1                ext4    defaults        0 0
/dev/mapper/vg18c-18cdb       /oradir2                ext4    defaults        0 0
[root@12r1-rac2 ~]# 
[root@12r1-rac2 ~]# 

If you would like to remove the LV, VG and PV, you can do the following
[root@12r1-rac2 /]# #Make sure to umount the file system. 
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cgrid
[root@12r1-rac2 /]# umount /dev/mapper/vg18c-18cdb
[root@12r1-rac2 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        25G  5.1G   19G  22% /
tmpfs           2.0G  633M  1.4G  32% /dev/shm
/dev/sda3       9.8G  5.9G  3.4G  64% /u01
/dev/sda2        12G  4.9G  6.3G  44% /u02
Oracle          500G  268G  233G  54% /media/sf_Oracle
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the logical volume
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cgrid
Do you really want to remove active logical volume 18cgrid? [y/n]: y
  Logical volume "18cgrid" successfully removed
[root@12r1-rac2 /]# lvremove /dev/mapper/vg18c-18cdb
Do you really want to remove active logical volume 18cdb? [y/n]: y
  Logical volume "18cdb" successfully removed
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#
[root@12r1-rac2 /]# #Remove the volume group
[root@12r1-rac2 /]# vgremove vg18c
  Volume group "vg18c" successfully removed
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# 
[root@12r1-rac2 /]# #Remove the Physical volumes created
[root@12r1-rac2 /]# pvremove /dev/sdf1 /dev/sdf2
  Labels on physical volume "/dev/sdf1" successfully wiped
  Labels on physical volume "/dev/sdf2" successfully wiped
[root@12r1-rac2 /]#
[root@12r1-rac2 /]#

References: