Exalogic Virtual Tea Break Snippets - Power Cycling Your Virtual Exalogic Rack

On rare occasions you may need to Power Cycle / Shutdown your Virtualised Exalogic Rack. Although the Power Cycle for the Physical Rack is documented in the "Exalogic Machine Owners Guide" and the Control Stack procedure is defined within a MOS note we do not have a single source that documents the Power Down / Power Up procedure for a fully Virtualised Exalogic Rack. This simple short blog entry will bring the distributed information into a single location and will also include the processes required for guest vServers.

At a High level the Power Cycle procedure is as follows:

Power Down

1. Shutdown all guest vServers.
2. Shutdown Control vServers (Control Stack).
3. Power down Physical Rack. 

Power Up

1. Power Up Physical Rack.
2. Start-up Control vServers
3. Start-up guest vServers.

The rest of this blog will go into more details about the high level steps above and draws from the "Exalogic Machine Owners Guide", "Exalogic Backup and Recovery Guide (MOS Note)" and the "Simple Exalogic Cli" blog entry to bring all the steps together in a single location.

Power Down

Guest Virtual Servers

Before shutting down the Control Virtual Machine we must ensure that all the Guest (Client) Virtual Servers have been stopped. This must be done using either EMOC or the associated IaaS CLI provided with EMOC. To facilitate a simple shutdown of all the Guest Virtual Server we can use the SimpleExaCli.sh wrapper provided in the "Simple Exaligic Cli blog entry. Once this wrapper script has been downloaded we can shutdown all Guest Virtual Servers are follows:

SimpleExCli.sh -u root -p wellknown --stop-vservers

The above command will find all Guest Virtual Servers defined in the Virtual Data Centre (vDC) and stop them. Therefore we must use a user that has access to all accounts within the vDC.

The root user is referenced in the calls to the SimpleExCli.sh script but this should be replaced by a Cloud User with access to all Accounts. If an appropriate Cloud User does not exist the the -a <Account> flag can be used to specify each account and the appropriate user/password used.

Control Virtual Servers (Control Stack)

Stopping the Exalogic Control Services

To stop the components of the Exalogic Control stack, do the following:

1. Log in to the Proxy Controller-2 virtual machine as the root user.
   Stop the Proxy Controller process.
   

   /opt/sun/xvmoc/bin/proxyadm stop -w

2. Log in to the Proxy Controller-1 virtual machine as the root user.
   Stop the Proxy Controller process.
   

   /opt/sun/xvmoc/bin/proxyadm stop -w

3. Log in to the Enterprise Controller virtual machine as the root user.
   Stop the Enterprise Controller process.
   

   /opt/sun/xvmoc/bin/ecadm stop -w

4. Log in to the Oracle VM Manager virtual machine as the root user.
   Stop the Oracle VM Manager process.
   

   service ovmm stop

5. Log in to the database virtual machine as the root user.
   Stop the database process.
   

   service oracle-db stop

This can be achieve with the following script (assume the system will not request a password).

#!/bin/sh

export EC1_IP=192.168.20.15
export PC1_IP=192.168.20.13
export PC2_IP=192.168.20.14
export OVMM_IP=192.168.20.11
export DB_IP=192.168.20.10

echo "********************************************"
echo STOPPING oc_pc1 ...
echo "********************************************"
echo
ssh root@$PC1_IP /opt/sun/xvmoc/bin/proxyadm stop -w
echo
echo "********************************************"
echo STOPPED oc_pc1 ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STOPPING oc_pc12 ...
echo "********************************************"
echo
ssh root@$PC2_IP /opt/sun/xvmoc/bin/proxyadm stop -w
echo
echo "********************************************"
echo STOPPED oc_pc2 ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STOPPING oc_ec1 ...
echo "********************************************"
echo
ssh root@$EC1_IP /opt/sun/xvmoc/bin/satadm stop -w
echo
echo "********************************************"
echo STOPPED oc_ec1 ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STOPPING ovmm ...
echo "********************************************"
echo
echo STOP ovmm ...
ssh root@$OVMM_IP service ovmm stop
echo
echo "********************************************"
echo STOPPED ovmm ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STOPPING oracle_db ...
echo "********************************************"
echo
ssh root@$DB_IP service oracle-db stop
echo
echo "********************************************"
echo STOPPED oracle_db ...
echo "********************************************"
echo
echo

Shutting Down the Control Virtual Machines

Gracefully shut down all Exalogic Control virtual machines by doing the following:

1. Log in to the Proxy Controller-2 virtual machine as the root user.
   Shut down the virtual machine, by running the following command:
   

   shutdown -h now

2. Log in to the Proxy Controller-1 virtual machine as the root user.
   Shut down the virtual machine, by running the following command:
   

   shutdown -h now

3. Log in to the Enterprise Controller virtual machine as the root user.
   Shut down the virtual machine, by running the following command:
   

   shutdown -h now

4. Log in to the Oracle VM Manager virtual machine as the root user.
   Shut down the virtual machine, by running the following command:
   

   shutdown -h now

5. Log in to the database virtual machine as the root user.
   Shut down the virtual machine, by running the following command:
   

   shutdown -h now

Physical Machine

1. Compute Nodes.
   

   shutdown -h -y now

2. Storage Server Heads.
3. Switches.
4. Rack.
   

Power Up

Physical Machine

1. Rack.
2. Switches.
   
   Ensure that the switches have had power applied for a few minutes to complete power-on configuration before starting the storage controllers and compute nodes
   
   
3. Storage Server Heads.
   
   Wait for a few minutes for the storage appliance to boot and start NFS services and daemons.
   
   
4. Compute Nodes.

After power is applied, the LEDs on all compute nodes and storage server heads will start blinking after a few minutes. From the rear of the rack, you can see the green LEDs on the PSUs turn on instantly after power is applied. In addition, from the rear of the rack, you can see the display on the PDUs that lights up once power is available.

Control Virtual Servers (Control Stack)

Starting the Control Virtual Machines

The Control Stack Virtual Machine must be started on the node they were originally running on and following a normal installation this will be as follows:

• Compute Node 1 :
• ExalogicControlDB
• ExalogicControlOVMM
• Compute Node 2 :
• ExalogicControlOpsCenterPC2
• Compute Node 3 :
• ExalogicControlOpsCenterPC1
• Compute Node 4 :
• ExalogicControlOpsCenterEC1

Before restarting we will need to identify the vm.cfg files associated with the Control vServers and this can be achieved as follows:

# cd /OVS/Repository//VirtualMachines
#grep -i ExalogicControl */vm.cfg

0004fb000006000038a23e7d5c307e00/vm.cfg:OVM_simple_name = 'ExalogicControlDB'
0004fb000006000056cafd150ebc34c0/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterPC2'
0004fb00000600009c3bf065d07004cb/vm.cfg:OVM_simple_name = 'ExalogicControlOVMM'
0004fb0000060000a5143f2ef7d264ac/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterPC1'
0004fb0000060000f0ccd3391c56a578/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterEC1'

To start the Exalogic Control Stack Virtual Machines we will use xm create on each node as follows:

1. Compute Node 1
   

   xm create <ExalogicControlDB Directory>/vm.cfg
   xm create <ExalogicControlOVMM Directory>/vm.cfg
   

2. Compute Node 4
   

   xm create <ExalogicControlEC1 Directory>/vm.cfg

3. Compute Node 2
   

   xm create <ExalogicControlPC2 Directory>/vm.cfg

4. Compute Node 3
   

   xm create <ExalogicControlPC1 Directory>/vm.cfg

Between each of the xm create commands we should pause to allow the Virtual Machine start-up to complete. If the services were stopped prior to shutting down the control stack, as described above, then on restart they will still be shutdown i.e. the state is retained. To restart the Control Stack Services you will need to follow the instructions below.

Starting the Exalogic Control Services

1. Log in to the database virtual machine as the root user.
   Start the database process.
   

   service oracle-db start

2. Log in to the Oracle VM Manager virtual machine as the root user.
   Start the Oracle VM Manager process.
   

   service ovmm start

3. Log in to the Enterprise Controller virtual machine as the root user.
   Start the Enterprise Controller process.
   

   /opt/sun/xvmoc/bin/ecadm start -w

4. Log in to the Proxy Controller-1 virtual machine as the root user.
   Start the Proxy Controller process.
   

   /opt/sun/xvmoc/bin/proxyadm start -w

5. Log in to the Proxy Controller-2 virtual machine as the root user.
   Start the Proxy Controller process.
   

   /opt/sun/xvmoc/bin/proxyadm start -w

This can be achieve with the following script (assume the system will not request a password).

#!/bin/sh

export EC1_IP=192.168.20.15
export PC1_IP=192.168.20.13
export PC2_IP=192.168.20.14
export OVMM_IP=192.168.20.11
export DB_IP=192.168.20.10


echo "********************************************"
echo STARTING oracle_db ...
echo "********************************************"
echo
ssh root@$DB_IP service oracle-db start
echo
echo "********************************************"
echo STARTED oracle_db ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STARTING ovmm ...
echo "********************************************"
echo
ssh root@$OVMM_IP service ovmm start
echo
echo "********************************************"
echo STARTED ovmm ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STARTING oc_ec1 ...
echo "********************************************"
echo
ssh root@$EC1_IP /opt/sun/xvmoc/bin/satadm start -w
echo
echo "********************************************"
echo STARTED oc_ec1 ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STARTING oc_pc2 ...
echo "********************************************"
echo
ssh root@$PC2_IP /opt/sun/xvmoc/bin/proxyadm start -w
echo
echo "********************************************"
echo STARTED oc_pc2 ...
echo "********************************************"
echo
echo

echo "********************************************"
echo STARTING oc_pc1 ...
echo "********************************************"
echo
ssh root@$PC1_IP /opt/sun/xvmoc/bin/proxyadm start -w
echo
echo "********************************************"
echo STARTED oc_pc1 ...
echo "********************************************"
echo
echo

Guest Virtual Servers

Before starting the Guest  Virtual Servers we should ensure that the Control Virtual Servers are running and this can be done by logging into EMOC and checking the status. Guest Virtual Server must only ever be started using either EMOC or the associate IaaS CLI (xm create must not be used). To facilitate a simplified interface the SimpleExCli.sh script, above, can be used to start all the Guest Virtual Servers within the Virtual Data Centre or just those in a specific Account.

SimpleExCli.sh -u root -p wellknown --start-vservers

The above command will find all Guest Virtual Servers defined in the Virtual Data Centre (vDC) and start them. Therefore we must use a user that has access to all accounts within the vDC. To start the Virtual Servers within a specific account we can execute.

SimpleExCli.sh -u root -p wellknown -a <Account Name> --start-vservers