In this post, we will look at how to create OS disks, discover and add/use disks to Oracle ASM storage without the installation of ASMLib package which would then install "oracleasm" command line interface.
We will be using UDEV rules in labeling the disks.
Environment used for this experiment is as below.
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production running on Enterprise Linux 6 (3.8.13-98.4.1.el6uek.x86_64) set up on a Oracle VM virtual box hosted on Windows 10 Enterprise.
I'm going to use a pen drive as an external storage (similar to a LUN added in real environment). The operations are given below which consists of comment lines explaining the purpose of each operation.
The below operations are done on the host machine (Windows 10 Enterprise)
We will be using UDEV rules in labeling the disks.
Environment used for this experiment is as below.
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production running on Enterprise Linux 6 (3.8.13-98.4.1.el6uek.x86_64) set up on a Oracle VM virtual box hosted on Windows 10 Enterprise.
I'm going to use a pen drive as an external storage (similar to a LUN added in real environment). The operations are given below which consists of comment lines explaining the purpose of each operation.
The below operations are done on the host machine (Windows 10 Enterprise)
-- From command prompt, cd to the pendrive directory (H:\ in my case)
H:\>dir
Volume in drive H has no label.
Volume Serial Number is 26D4-C78B
Directory of H:\
03/03/2016 11:01 AM <DIR> ASM_DISK
0 File(s) 0 bytes
1 Dir(s) 1,477,132,288 bytes free
H:\>cd ASM_DISK
H:\ASM_DISK>VBoxManage createhd --filename asm5.vdi --size 3072 --format VDI --variant Fixed
'VBoxManage' is not recognized as an internal or external command,
operable program or batch file.
-- Use full path or set the environment variable before launching the VBoxManage command.
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm5.vdi --size 3072 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: f71e63e5-e3cd-4ef6-afbd-5472fd3315fb
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm6.vdi --size 3072 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 205a7c85-6b50-45c5-9f6e-5b3156f53095
-- Add the storage to the Virtual Box host machine and make it sharable to be used for RAC.
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach 12r1-rac1 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable
VBoxManage.exe: error: The machine is not mutable (state is Running)
VBoxManage.exe: error: Details: code VBOX_E_INVALID_VM_STATE (0x80bb0002), component SessionMachine, interface IMachine, callee IUnknown
VBoxManage.exe: error: Context: "COMSETTER(PortCount)(port + 1)" at line 385 of file VBoxManageStorageController.cpp
VBoxManage.exe: error: The port and/or device parameter are out of range: port=5 (must be in range [0, 4]), device=0 (must be in range [0, 0])
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component StorageControllerWrap, interface IStorageController, callee IUnknown
VBoxManage.exe: error: Context: "AttachDevice(Bstr(pszCtl).raw(), port, device, DeviceType_HardDisk, pMedium2Mount)" at line 758 of file VBoxManageStorageController.cpp
-- We can't add the storage to the guest machine when it's running. Shut down the machine and try again.
-- I've shut the machine down and retrying the operation.
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach 12r1-rac1 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach 12r1-rac2 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm5.vdi --type shareable
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach 12r1-rac1 --storagectl "SATA" --port 6 --device 0 --type hdd --medium asm6.vdi --mtype shareable
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach 12r1-rac2 --storagectl "SATA" --port 6 --device 0 --type hdd --medium asm6.vdi --mtype shareable
H:\ASM_DISK>"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm6.vdi --type shareable
Now let's continue the rest of the tasks. Note that we will have sudo access in real environment but here I'm directly login as root as this is a test machine.
login as: root
root@192.168.56.101's password:
Last login: Mon Nov 2 00:14:18 2015
[root@12r1-rac1 ~]# cd /dev
[root@12r1-rac1 dev]# ls -l sd*
brw-rw----. 1 root disk 8, 0 Mar 3 23:39 sda
brw-rw----. 1 root disk 8, 1 Mar 3 23:39 sda1
brw-rw----. 1 root disk 8, 2 Mar 3 23:39 sda2
brw-rw----. 1 root disk 8, 3 Mar 3 23:39 sda3
brw-rw----. 1 root disk 8, 4 Mar 3 23:39 sda4
brw-rw----. 1 root disk 8, 5 Mar 3 23:39 sda5
brw-rw----. 1 root disk 8, 16 Mar 3 23:39 sdb
brw-rw----. 1 root disk 8, 32 Mar 3 23:39 sdc
brw-rw----. 1 root disk 8, 48 Mar 3 23:39 sdd
brw-rw----. 1 root disk 8, 64 Mar 3 23:39 sde
brw-rw----. 1 root disk 8, 80 Mar 3 23:39 sdf
brw-rw----. 1 root disk 8, 96 Mar 3 23:39 sdg
-- As I already had partition till sde, I'm now going to create partitions for sdf and sdg devices. As the device is sharable, it is required to do the partition on any of the node in the RAC cluster.
[root@12r1-rac1 dev]# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xaa4dc9c9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-391, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-391, default 391):
Using default value 391
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@12r1-rac1 dev]# fdisk /dev/sdg
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x6b321ecc.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-391, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-391, default 391):
Using default value 391
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
-- Now "fdisk -l" on both the nodes will show the partions created.
-- scsi_id.config should be updated with options=-g and it's already set up
-- Find the disk scsi ids of the added disks. (Line 86 and 88)
-- Add udev rules for the devices and label them (Lines 101 and 102)
[root@12r1-rac1 dev]# more /etc/scsi_id.config
options=-g
[root@12r1-rac1 dev]# /sbin/scsi_id -g -u -d /dev/sdf
1ATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd
[root@12r1-rac1 dev]# /sbin/scsi_id -g -u -d /dev/sdg
1ATA_VBOX_HARDDISK_VB205a7c85-9530f556
[root@12r1-rac1 dev]# more /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB61c71f2c-051fa77b", NAME="asm-disk1", OWNER="oracle", GROUP="
dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBc35ab459-42e9ac5a", NAME="asm-disk2", OWNER="oracle", GROUP="
dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB4db5a8cc-189477d6", NAME="asm-disk3", OWNER="oracle", GROUP="
dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBfeeeb790-dd25f13b", NAME="asm-disk4", OWNER="oracle", GROUP="
dba", MODE="0660"
[root@12r1-rac1 dev]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
-- Add the following lines
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd", NAME="asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB205a7c85-9530f556", NAME="asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"
[root@12r1-rac1 dev]# /sbin/partprobe /dev/sdf1
[root@12r1-rac1 dev]# /sbin/partprobe /dev/sdg1
On second node, manual "/sbin/partprobe" has to be done to find the devices. You can see all the partitioned disks once partprobe is done.I have shown the ASM device rules and testing for the same for node 2 but it should be done on both the nodes.
[root@12r1-rac2 oracle]# /sbin/partprobe [root@12r1-rac2 oracle]# cat /proc/partitions major minor #blocks name 11 0 1048575 sr0 8 0 52428800 sda 8 1 26214400 sda1 8 2 12579840 sda2 8 3 10485760 sda3 8 4 1 sda4 8 5 3145728 sda5 8 16 5242880 sdb 8 17 5237158 sdb1 8 32 5242880 sdc 8 33 5237158 sdc1 8 48 5242880 sdd 8 49 5237158 sdd1 8 64 5242880 sde 8 65 5237158 sde1 8 80 3145728 sdf 8 81 3140676 sdf1 8 96 3145728 sdg 8 97 3140676 sdg1 [root@12r1-rac2 oracle]# /sbin/partprobe /dev/sdf1 [root@12r1-rac2 oracle]# /sbin/partprobe /dev/sdg1 [root@12r1-rac2 oracle]# -- Edit the ASM device rules and test for the rules working on both the nodes. [root@12r1-rac2 oracle]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules [root@12r1-rac2 oracle]# udevadm test /block/sdf/sdf1 [root@12r1-rac2 oracle]# udevadm test /block/sdg/sdg1 [root@12r1-rac2 oracle]# -- Reload rules and start the UDEV service on both the nodes. [root@12r1-rac2 oracle]# udevadm control --reload-rules [root@12r1-rac2 oracle]# /sbin/start_udev Starting udev: [ OK ] -- At this point, we are able to see the disks successfully [root@12r1-rac2 oracle]#cd /dev [root@12r1-rac2 dev]# ls -al asm* brw-rw----. 1 oracle dba 8, 17 Mar 4 10:17 asm-disk1 brw-rw----. 1 oracle dba 8, 33 Mar 4 10:16 asm-disk2 brw-rw----. 1 oracle dba 8, 49 Mar 4 10:17 asm-disk3 brw-rw----. 1 oracle dba 8, 65 Mar 4 10:17 asm-disk4 brw-rw----. 1 oracle dba 8, 81 Mar 4 10:14 asm-disk5 brw-rw----. 1 oracle dba 8, 97 Mar 4 10:15 asm-disk6Below are the excerpts from the udevadm test result where the highlighted lines show the successful labelling...
... ... ... ... util_run_program: '/sbin/scsi_id' (stderr) 'scsi_id[16165]: custom logging function 0x7fcf4271f010 registered' util_run_program: '/sbin/scsi_id' (stdout) '1ATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd' util_run_program: '/sbin/scsi_id -g -u -d /dev/sdf' returned with exitcode 0 udev_rules_apply_to_event: OWNER 54321 /etc/udev/rules.d/99-oracle-asmdevices.rules:5 udev_rules_apply_to_event: GROUP 54322 /etc/udev/rules.d/99-oracle-asmdevices.rules:5 udev_rules_apply_to_event: MODE 0660 /etc/udev/rules.d/99-oracle-asmdevices.rules:5 udev_rules_apply_to_event: NAME 'asm-disk5' /etc/udev/rules.d/99-oracle-asmdevices.rules:5 udev_rules_apply_to_event: PROGRAM '/sbin/scsi_id -g -u -d /dev/sdf' /etc/udev/rules.d/99-oracle-asmdevices.rules:6 util_run_program: '/sbin/scsi_id -g -u -d /dev/sdf' started util_run_program: '/sbin/scsi_id' (stderr) 'scsi_id[16166]: custom logging function 0x7f9e6b491010 registered' util_run_program: '/sbin/scsi_id' (stdout) '1ATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd' util_run_program: '/sbin/scsi_id -g -u -d /dev/sdf' returned with exitcode 0 udev_device_update_db: created db file for '/devices/pci0000:00/0000:00:0d.0/ata8/host7/target7:0:0/7:0:0:0/block/sdf/sdf1' in '/dev/.udev/db/block:sdf1' udev_node_add: creating device node '/dev/asm-disk5', devnum=8:81, mode=0660, uid=54321, gid=54322 udev_node_mknod: preserve file '/dev/asm-disk5', because it has correct dev_t udev_node_mknod: preserve permissions /dev/asm-disk5, 060660, uid=54321, gid=54322 node_symlink: preserve already existing symlink '/dev/block/8:81' to '../asm-disk5' link_find_prioritized: found '/sys/devices/pci0000:00/0000:00:0d.0/ata8/host7/target7:0:0/7:0:0:0/block/sdf/sdf1' claiming '/dev/.udev/links/disk\x2fby-id\x2fata-VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' link_update: creating link '/dev/disk/by-id/ata-VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' to '/dev/asm-disk5' node_symlink: preserve already existing symlink '/dev/disk/by-id/ata-VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' to '../../asm-disk5' link_find_prioritized: found '/sys/devices/pci0000:00/0000:00:0d.0/ata8/host7/target7:0:0/7:0:0:0/block/sdf/sdf1' claiming '/dev/.udev/links/disk\x2fby-id\x2fscsi-SATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' link_update: creating link '/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' to '/dev/asm-disk5' node_symlink: preserve already existing symlink '/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBf71e63e5-fb1533fd-part1' to '../../asm-disk5' link_find_prioritized: found '/sys/devices/pci0000:00/0000:00:0d.0/ata9/host8/target8:0:0/8:0:0:0/block/sdg/sdg1' claiming '/dev/.udev/links/disk\x2fby-path\x2fpci-0000:00:0d.0-scsi-0:0:0:0-part1' ... ... ... ... udev_device_update_db: created db file for '/devices/pci0000:00/0000:00:0d.0/ata9/host8/target8:0:0/8:0:0:0/block/sdg/sdg1' in '/dev/.udev/db/block:sdg1' udev_node_add: creating device node '/dev/asm-disk6', devnum=8:97, mode=0660, uid=54321, gid=54322 udev_node_mknod: preserve file '/dev/asm-disk6', because it has correct dev_t udev_node_mknod: preserve permissions /dev/asm-disk6, 060660, uid=54321, gid=54322 node_symlink: preserve already existing symlink '/dev/block/8:97' to '../asm-disk6' link_find_prioritized: found '/sys/devices/pci0000:00/0000:00:0d.0/ata9/host8/target8:0:0/8:0:0:0/block/sdg/sdg1' claiming '/dev/.udev/links/disk\x2fby-id\x2fata-VBOX_HARDDISK_VB205a7c85-9530f556-part1' link_update: creating link '/dev/disk/by-id/ata-VBOX_HARDDISK_VB205a7c85-9530f556-part1' to '/dev/asm-disk6' node_symlink: preserve already existing symlink '/dev/disk/by-id/ata-VBOX_HARDDISK_VB205a7c85-9530f556-part1' to '../../asm-disk6' link_find_prioritized: found '/sys/devices/pci0000:00/0000:00:0d.0/ata9/host8/target8:0:0/8:0:0:0/block/sdg/sdg1' claiming '/dev/.udev/links/disk\x2fby-id\x2fscsi-SATA_VBOX_HARDDISK_VB205a7c85-9530f556-part1' link_update: creating link '/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VB205a7c85-9530f556-part1' to '/dev/asm-disk6' node_symlink: preserve already existing symlink '/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VB205a7c85-9530f556-part1' to '../../asm-disk6' ... ... ...Now we have done with labelling the disks so that Oracle can use it. Next step would be to create a ASM disk group using these disks that were added which is as shown below.
[oracle@12r1-rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.1.0.1.0 Production on Thu Mar 24 19:47:27 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set lines 300
SQL> col path for a30
SQL> -- See that the added disks are shown with HEADER STATUS as CANDIDATE.
SQL> select GROUP_NUMBER, DISK_NUMBER,MOUNT_STATUS, STATE,HEADER_STATUS,TOTAL_MB, FREE_MB, NAME,PATH from v$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S STATE HEADER_STATU TOTAL_MB FREE_MB NAME PATH
------------ ----------- ------- -------- ------------ ---------- ---------- ------------------------------ ------------------------------
0 0 CLOSED NORMAL CANDIDATE 0 0 /dev/asm-disk6
0 1 CLOSED NORMAL CANDIDATE 0 0 /dev/asm-disk5
1 2 CACHED NORMAL MEMBER 5114 3047 DATA_0002 /dev/asm-disk3
1 0 CACHED NORMAL MEMBER 5114 3024 DATA_0000 /dev/asm-disk1
1 3 CACHED NORMAL MEMBER 5114 3051 DATA_0003 /dev/asm-disk4
1 1 CACHED NORMAL MEMBER 5114 3052 DATA_0001 /dev/asm-disk2
6 rows selected.
SQL> CREATE DISKGROUP DATA2 EXTERNAL REDUNDANCY DISK
2 '/dev/asm-disk5' NAME DG2_1,
3 '/dev/asm-disk6' NAME DG2_2;
Diskgroup created.
SQL> select GROUP_NUMBER, DISK_NUMBER,MOUNT_STATUS, STATE,HEADER_STATUS,TOTAL_MB, FREE_MB, NAME,PATH from v$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S STATE HEADER_STATU TOTAL_MB FREE_MB NAME PATH
------------ ----------- ------- -------- ------------ ---------- ---------- ------------------------------ ------------------------------
1 2 CACHED NORMAL MEMBER 5114 3047 DATA_0002 /dev/asm-disk3
1 0 CACHED NORMAL MEMBER 5114 3024 DATA_0000 /dev/asm-disk1
1 3 CACHED NORMAL MEMBER 5114 3051 DATA_0003 /dev/asm-disk4
1 1 CACHED NORMAL MEMBER 5114 3052 DATA_0001 /dev/asm-disk2
2 0 CACHED NORMAL MEMBER 3067 3040 DG2_1 /dev/asm-disk5
2 1 CACHED NORMAL MEMBER 3067 3042 DG2_2 /dev/asm-disk6
6 rows selected.
SQL>
Mission accomplished!Update: Identifying disk ids has been changed in RHEL 7
Use the code like below as the location of scsi_id has been changed
For eg:
[root@xxx]# /lib/udev/scsi_id -g -u -d /dev/sdt
36000c29347de4ab25c589e591dbe4dd9
[root@xxx]# /lib/udev/scsi_id -g -u -d /dev/sdu
36000c2961530dd7b50a54689da6c6641
Happy ASM disking! :)
