Gotcha Adding Disk to Virtual Oracle RAC Cluster with VMware multi-writer flag

While I was working on the Oracle Databases on Web Scale Tech Note for Nutanix I needed a way to quickly add disks to my Oracle RAC VM’s and set the multi-writer flag at the same time. To do this I created a PowerCLI script. It’s quite simple in what it does, but it works, and met my requirements. In the process however I ran into a VMware bug as well. So in this article I’ll explain the bug, and include my script and how it works. You can use this to add disks to an Oracle RAC node and set the multi-writer flag.

Take a look at the Oracle Databases on Web-Scale Infrastructure Tech Note and you will see some of what I’ve been up to lately. Including vMotion of Monster Oracle RAC VM’s on the Nutanix platform. This document contains some of my secret source that I’ve used for tuning systems for over a decade. I’m working on another paper that goes into more depth and contains even more of the tuning secret source. I will let you know when that is available.

As I was going through the work for this paper I was building up a large Oracle RAC system of about 4TB in total size on 3 Oracle RAC 12c Nodes (8 vCPU, 112GB RAM each). I found that I couldn’t add a virtual disk to the Oracle RAC cluster while the VM’s are running. I was running all my tests on vSphere 5.5. I knew from previous experience that at one time this worked on previous releases. You were able to add disks to live running Oracle RAC clusters with the multi-writer flag before.

But in 5.5 this has changed and it is not possible to add a disk to a live running VM due to a change in the VIM API. If you attempt it the operation will fail complaining that the disk is locked. There is no KB article on this issue, although VMware Engineering / Support is aware of this problem. This is sort of a regression. But in some ways I think maybe it was an accident or unintended that it worked before and in 5.5 it’s been ‘fixed’.

Whatever the case the only solution at the moment is to add the disks to the nodes when they are offline. At least it can be done very quickly, especially when using PowerCLI. So your outage window will be very limited.  Unfortunately as all the disks are shared that means all of the Oracle RAC Nodes need to be shut down at the same time. Compared to the benefits of virtualizing Oracle RAC this is a minor inconvenience. Hopefully VMware will fix this in an upcoming update, or that you aren’t using 5.5 yet for Oracle RAC (5.0 and 5.1 still work as far as I know, I haven’t tested the recent update releases though). Also don’t try and extend the disks of the running VM’s while they’re operational, that is also not supported and won’t work. If you’re using DNFS, or in guest storage access to another storage system these restrictions may not apply.

Here are the simple scripts that I mocked up to add the disks to my Oracle RAC nodes. These quite basic and I can think of lots of ways to improve them, including taking input for more than one operation at a time. But these are a start. If you choose to use them and improve upon them I’d be happy to hear about it and would like to add the improvements here. So please let me know how you get on. Please bare in mind that these scripts are without warranty of any kind and for use at your own risk and informational purposes only. I’m hoping in the future it’ll be easy to set a disk to multi-writer when it’s created through the New-HardDisk CmdLet, without using the VIM API. This would make the scripts significantly simpler.

Add New Disk to Oracle RAC Node with Multi-writer Flag set:

Connect-VIServer <YourvCenter>

$vmname = Read-Host “VM Name to add disks to”

$vmdk = Read-Host “Disk to Add (Full Path)”

[int]$busnumber = Read-host “SCSI Bus Number Number”

[int]$unitnumber = Read-Host “Disk Unit Number”

[int]$capacity = Read-Host “Disk Capacity (GB)”

$vm = Get-VM $vmname

[int64]$capcitykb = ($capacity * 1gb) / 1kb

Write-Host “Creating New Disk ” $vmdk ” of size ” $capacity ” GB to ” $vm ” at SCSI ID ” $busnumber “:” $unitnumber

Write-Host “Disk Capacity in KB ” $capcitykb

$ctrll = Get-ScsiController -VM $vm | ?{$_.extensiondata.busNumber -eq $busnumber}

Write-Host “Controller Key “$ctrll.extensiondata.key

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (1)

$spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

$spec.deviceChange[0].operation = “add”

$spec.deviceChange[0].fileOperation = “create”

$spec.deviceChange[0].device = New-Object VMware.Vim.VirtualDisk

$spec.deviceChange[0].device.key = -100

$spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualDiskFlatVer2BackingInfo

$spec.deviceChange[0].device.backing.fileName = “”+$vmdk

$spec.deviceChange[0].device.backing.thinProvisioned = $false

$spec.deviceChange[0].device.backing.split = $false

$spec.deviceChange[0].device.backing.writeThrough = $false

$spec.deviceChange[0].device.backing.eagerlyScrub = $true

$spec.deviceChange[0].device.backing.diskMode = “independent_persistent”

$spec.deviceChange[0].device.connectable = New-Object VMware.Vim.VirtualDeviceConnectInfo

$spec.deviceChange[0].device.connectable.startConnected = $true

$spec.deviceChange[0].device.connectable.allowGuestControl = $false

$spec.deviceChange[0].device.connectable.connected = $true

$spec.deviceChange[0].device.controllerKey = [int]$ctrll.extensiondata.key

$spec.deviceChange[0].device.unitNumber = [int]$unitNumber

$spec.deviceChange[0].device.capacityInKB = [int64]$capacitykb

$spec.extraConfig = New-Object VMware.Vim.OptionValue[] (1)

$spec.extraConfig[0] = New-Object VMware.Vim.OptionValue

$spec.extraConfig[0].key = “scsi” + $busnumber + “:” + $unitnumber + “.sharing”

$spec.extraConfig[0].value = “multi-writer”

$vm.ExtensionData.ReconfigVM($spec)

Add Existing Disk to Oracle RAC Node with Multi-writer Flag Set:

Connect-VIServer <YourvCenterHere>

$vmname = Read-Host “VM Name to add disks to”

$vmdk = Read-Host “Disk to Add”

[int]$busnumber = Read-host “SCSI Bus Number Number”

[int]$unitnumber = Read-Host “Disk Unit Number”

$vm = Get-VM $vmname

Write-Host “Adding Disk “$vmdk” to “$vm” at SCSI ID “$busnumber”:”$unitnumber

$ctrll = Get-ScsiController -VM $vm | ?{$_.extensiondata.busNumber -eq $busnumber}

Write-Host “Controller Key “$ctrll.extensiondata.key

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (1)

$spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

$spec.deviceChange[0].operation = “add”

$spec.deviceChange[0].device = New-Object VMware.Vim.VirtualDisk

$spec.deviceChange[0].device.key = -100

$spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualDiskFlatVer2BackingInfo

$spec.deviceChange[0].device.backing.fileName = “”+$vmdk

$spec.deviceChange[0].device.backing.diskMode = “independent_persistent”

$spec.deviceChange[0].device.connectable = New-Object VMware.Vim.VirtualDeviceConnectInfo

$spec.deviceChange[0].device.connectable.startConnected = $true

$spec.deviceChange[0].device.connectable.allowGuestControl = $false

$spec.deviceChange[0].device.connectable.connected = $true

$spec.deviceChange[0].device.controllerKey = [int]$ctrll.extensiondata.Key

$spec.deviceChange[0].device.unitNumber = [int]$unitNumber

$spec.extraConfig = New-Object VMware.Vim.OptionValue[] (1)

$spec.extraConfig[0] = New-Object VMware.Vim.OptionValue

$spec.extraConfig[0].key = “scsi” + $busnumber + “:” + $unitnumber + “.sharing”

$spec.extraConfig[0].value = “multi-writer”

$vm.ExtensionData.ReconfigVM($spec)

Final Word

Hopefully these simple scripts will help you when creating net new Oracle RAC clusters on VMware vSphere and when adding disks to existing clusters. You can use them for your own purposes and modify them as you want. Additional information can be found on the VMware Business Critical Apps Blog.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster. All rights reserved. Not to be reproduced for commercial purposes without written permission.