Thursday, 1 March 2012

Exalogic Networking Part 2

Following on from my introduction Exalogic Blog Entry (Exalogic Networking Part 1) this entry describes the Oracle A-Team Exalogic Scripts to help facilitate a simpler ECU installation and provide an example on configuring a standard Oracle ACS type environment. The scripts and a pdf document can be downloaded using the link below :

This blog will summarise the functionality that is provided by the A-Team Exalogic Scripts. These expand upon the existing “Exalogic Configuration Utility” functionality simplifying repetitive jobs, for example creating VNICs on the switches, and building base environments.

Version 1.0.1 contains the following scripts that will be detailed later:
During the process of execution the scripts above will generate additional scripts that will be specific for Exalogic Rack they are being executed on.

Switch Scripts

This script in itself does not generate any additional scripts rather it will interrogate the Switch it is running one to identify which, if any, of the 10 Gb connections are connected. This is achieved by checking the Bridge Status
Connector 0A-ETH Present
  Bridge-0 Port 0A-ETH-1 (Bridge-0-2) down (Enabled)
  Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
  Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
  Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Connector 1A-ETH Present
  Bridge-1 Port 1A-ETH-1 (Bridge-1-2) up (Enabled)
  Bridge-1 Port 1A-ETH-2 (Bridge-1-2) up (Enabled)
  Bridge-1 Port 1A-ETH-3 (Bridge-1-1) up (Enabled)
  Bridge-1 Port 1A-ETH-4 (Bridge-1-1) up (Enabled)
The Bridge Connections with a status of up will be extracted and the ETH Port used in repeated calls to the The net result of this is that the script will generate a number of “” files based on the switch name and the appropriate port as explain in the


./ [-f listlinkup output] [-i ibnetdiscover output] [-v showvnics output]


-f : This the location of a file containing the output from the listlinkup command. If one is not provided then the script will execute this command and save the file.
-I : This is the location containing the output from the ibnetdiscover command . If no file is provided then the script will execute ibnetdiscover and save the file.
-v : This is the location containing the output from the showvnics  command . If no file is provided then the script will execute showvnics  and save the file.

The purpose of this script is to generate a one or more executable scripts that will create the appropriate VNICs on the Switch. By default the script will only generate files relevant to the switch it is being executed on but the user can optionally (using the –a flag) generate files for all discoverable switches. Once the script has been executed for each switch / ETH Port combination two files will be generated:

  1. create-<Switch Name>-<Eth Port>
  2. delete-<Switch Name>-<Eth Port>
These generated scripts can then be used to create / remove the vnics on the specified switch.

Create-<Switch Name>-<ETH Port>

# "el2bcn15 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:f5:fa -mac a0:f5:fa:10:00:67 -pkey default
# "el2bcn16 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:fb:4a -mac a0:fb:4a:10:00:68 -pkey default
# "el2bcn08 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d5:85 -mac a0:d5:85:10:00:58 -pkey default
# "el2bcn09 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d6:fe -mac a0:d6:fe:10:00:59 -pkey default
# "el2bcn12 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d5:6e -mac a0:d5:6e:10:00:62 -pkey default
# "el2bcn11 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d7:ea -mac a0:d7:ea:10:00:61 -pkey default
# "el2bcn14 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a1:0c:0e -mac a0:0c:0e:10:00:64 -pkey default
# "el2bcn13 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d4:42 -mac a0:d4:42:10:00:63 -pkey default
# "el2bcn10 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:ed:c6 -mac a0:ed:c6:10:00:60 -pkey default
# "el2bcn07 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:da:15 -mac a0:da:15:10:00:57 -pkey default
# "el2bcn05 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:c3:59 -mac a0:c3:59:10:00:55 -pkey default
# "el2bcn06 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:da:69 -mac a0:da:69:10:00:56 -pkey default
# "el2bcn03 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d6:59 -mac a0:d6:59:10:00:53 -pkey default
# "el2bcn04 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d8:45 -mac a0:d8:45:10:00:54 -pkey default
# "el2bcn01 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d6:f9 -mac a0:d6:f9:10:00:51 -pkey default
# "el2bcn02 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d7:c5 -mac a0:d7:c5:10:00:52 -pkey default


ID  STATE     FLG IOA_GUID                NODE       IID  MAC               VLN PKEY   GW
--- --------  --- ----------------------- ---------- ---- ----------------- --- ----   --------
 12 UP          N 00:21:28:00:01:A1:0C:0E el2bcn14 EL-C 0000 A0:0C:0E:50:00:14 NO  ffff   1A-ETH-1
 15 UP          N 00:21:28:00:01:A0:DA:15 el2bcn07 EL-C 0000 A0:DA:15:50:00:07 NO  ffff   1A-ETH-1
 13 UP          N 00:21:28:00:01:A0:D4:42 el2bcn13 EL-C 0000 A0:D4:42:50:00:13 NO  ffff   1A-ETH-1
 19 UP          N 00:21:28:00:01:A0:D8:45 el2bcn04 EL-C 0000 A0:D8:45:50:00:04 NO  ffff   1A-ETH-1
  7 UP          N 00:21:28:00:01:A0:FB:4A el2bcn16 EL-C 0000 A0:FB:4A:50:00:18 NO  ffff   1A-ETH-1
 16 UP          N 00:21:28:00:01:A0:C3:59 el2bcn05 EL-C 0000 A0:C3:59:50:00:05 NO  ffff   1A-ETH-1
 18 UP          N 00:21:28:00:01:A0:D6:59 el2bcn03 EL-C 0000 A0:D6:59:50:00:03 NO  ffff   1A-ETH-1
 17 UP          N 00:21:28:00:01:A0:DA:69 el2bcn06 EL-C 0000 A0:DA:69:50:00:06 NO  ffff   1A-ETH-1
 10 UP          N 00:21:28:00:01:A0:D5:6E el2bcn12 EL-C 0000 A0:D5:6E:50:00:12 NO  ffff   1A-ETH-1
  8 UP          N 00:21:28:00:01:A0:D5:85 el2bcn08 EL-C 0000 A0:D5:85:50:00:08 NO  ffff   1A-ETH-1
 21 UP          N 00:21:28:00:01:A0:D7:C5 el2bcn02 EL-C 0000 A0:D7:C5:50:00:02 NO  ffff   1A-ETH-1
 14 UP          N 00:21:28:00:01:A0:ED:C6 el2bcn10 EL-C 0000 A0:ED:C6:50:00:10 NO  ffff   1A-ETH-1
 11 UP          N 00:21:28:00:01:A0:D7:EA el2bcn11 EL-C 0000 A0:D7:EA:50:00:11 NO  ffff   1A-ETH-1
 20 UP          N 00:21:28:00:01:A0:D6:F9 el2bcn01 EL-C 0000 A0:D6:F9:50:00:51 NO  ffff   1A-ETH-1
  6 UP          N 00:21:28:00:01:A0:F5:FA el2bcn15 EL-C 0000 A0:F5:FA:50:00:17 NO  ffff   1A-ETH-1
  9 UP          N 00:21:28:00:01:A0:D6:FE el2bcn09 EL-C 0000 A0:D6:FE:50:00:09 NO  ffff   1A-ETH-1


./ [-f ibnetdiscover output] [-p Ethernet Port (eg 0A-ETH-1)] [-a generate for all visible switches] [-v showvnics output] [-c clear /conf/bx.conf]


-f : This is the location containing the output from the ibnetdiscover command . If no file is provided then the script will execute ibnetdiscover and save the file.
-p : The ETH Port to which the VNICs will be bond.
-a : Flag to indicate that files should be generated for all switches that are discoverable.
-v : This is the location containing the output from the showvnics  command . If no file is provided then the script will execute showvnics  and save the file. This is used to generate the appropriate deletevnic command for all vnics on the specified port.
-c : Flag to indicate if the /conf/bx.conf file should be re-initialised.

Compute Node Scripts

On execution of this script it will generate the ifcfg-eth4, ifcfg-eth5 and ifcfg-bond1 scripts on the node for the given parameters, edit the /etc/sysconfig/network file and optionally modify the /etc/sysctl.conf to set the rp_filter to loose. If any of these scripts already exist they will be backed up in the same directory to allow a manual rollback. Once the files have been created the script will execute a “service network restart” to restart the network and hence force a re-read of the information. At this point a executing netstat –rn should show that the bond1 is the default gateway.

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface   U         0 0          0 bond1   U         0 0          0 bond0     U         0 0          0 eth0     U         0 0          0 eth0         UG        0 0          0 bond1
If the bond1 (10Gb) network is not default then you will need to delete the default route and add the 10 Gb bond1 as the default.






BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"




./ -ip <Bond 1 (10 Gb Eth) IP Address> -nm <Bond 1 (10 Gb Eth) Net Mask> -gw <Bond 1 (10 Gb Eth) Gateway> -m4 <Eth 4 Hardware Mac Address (Primary)> -m5 <Eth 5 Hardware Mac Address (Secondary)>


-ip : The 10 Gb IP Address that should be assigned to bond1.
-nm : The Netmask associated with the 10 Gb network.
-gw : The gateway address associated with the 10 Gb bond1 network.
-m4 : The Eth 4 MAC Address as generated by the
-m5 : The Eth 5 MAC Address as generated by the

This file does not affect the system directly instead it will generate a file “” that can be executed to configure each node (example below). The generated file assumes that exists in the same directory. The generated file, when executed, will initially recreate the trust relationship, deleted at the end of the Exalogic Configuration Utility, between the nodes (unless –I is specified) and then copy (using dcli) the to each of the nodes and then execute it on each of the nodes. This execution is not done using dcli because each node has a specific configuration that requires a direct call using ssh.
In addition a “” will be created that can be executed via dcli to reconfigure the routing table to use the 10Gb (bond1) gateway if this was not successfully done during the execute of and its in build “service network restart”.


while [ $# -gt 0 ]
 case "$1" in 
  -i) SETUPSSHTRUST=false;;
  *) echo ""; echo >&2 \
      "usage: $0 [-i ]"
      echo""; exit 1;;
  *) break;;

if test "$SETUPSSHTRUST" = "true"
 echo "Setting up SSH Trust"
 echo ""
 ssh-keygen -t dsa
 /opt/ -k -g nodelist 

chmod +x
/opt/ -g nodelist mkdir -p /tmp/GenScripts
/opt/ -g nodelist -d /tmp/GenScripts -f
/opt/ -g nodelist -d /tmp/GenScripts -f

# Call Script on el2bcn01
ssh -l root el2bcn01 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D6:F9:50:00:51 -m5 A0:D6:FA:50:00:51 -cn el2bcn01
# Call Script on el2bcn02
ssh -l root el2bcn02 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D7:C5:50:00:02 -m5 A0:D7:C6:50:00:02 -cn el2bcn02
# Call Script on el2bcn03
ssh -l root el2bcn03 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D6:59:50:00:03 -m5 A0:D6:5A:50:00:03 -cn el2bcn03
# Call Script on el2bcn04
ssh -l root el2bcn04 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D8:45:50:00:04 -m5 A0:D8:46:50:00:04 -cn el2bcn04
# Call Script on el2bcn05
ssh -l root el2bcn05 /tmp/GenScripts/ -ip -nm -gw -m4 A0:C3:59:50:00:05 -m5 A0:C3:5A:50:00:05 -cn el2bcn05
# Call Script on el2bcn06
ssh -l root el2bcn06 /tmp/GenScripts/ -ip -nm -gw -m4 A0:DA:69:50:00:06 -m5 A0:DA:6A:50:00:06 -cn el2bcn06
# Call Script on el2bcn07
ssh -l root el2bcn07 /tmp/GenScripts/ -ip -nm -gw -m4 A0:DA:15:50:00:07 -m5 A0:DA:16:50:00:07 -cn el2bcn07
# Call Script on el2bcn08
ssh -l root el2bcn08 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D5:85:50:00:08 -m5 A0:D5:86:50:00:08 -cn el2bcn08
# Call Script on el2bcn09
ssh -l root el2bcn09 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D6:FE:50:00:09 -m5 A0:D6:FD:50:00:09 -cn el2bcn09
# Call Script on el2bcn10
ssh -l root el2bcn10 /tmp/GenScripts/ -ip -nm -gw -m4 A0:ED:C6:50:00:10 -m5 A0:ED:C5:50:00:10 -cn el2bcn10
# Call Script on el2bcn11
ssh -l root el2bcn11 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D7:EA:50:00:11 -m5 A0:D7:E9:50:00:11 -cn el2bcn11
# Call Script on el2bcn12
ssh -l root el2bcn12 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D5:6E:50:00:12 -m5 A0:D5:6D:50:00:12 -cn el2bcn12
# Call Script on el2bcn13
ssh -l root el2bcn13 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D4:42:50:00:13 -m5 A0:D4:41:50:00:13 -cn el2bcn13
# Call Script on el2bcn14
ssh -l root el2bcn14 /tmp/GenScripts/ -ip -nm -gw -m4 A0:0C:0E:50:00:14 -m5 A0:0C:0D:50:00:14 -cn el2bcn14
# Call Script on el2bcn15
ssh -l root el2bcn15 /tmp/GenScripts/ -ip -nm -gw -m4 A0:F5:FA:50:00:17 -m5 A0:F5:F9:50:00:17 -cn el2bcn15
# Call Script on el2bcn16
ssh -l root el2bcn16 /tmp/GenScripts/ -ip -nm -gw -m4 A0:FB:4A:50:00:18 -m5 A0:FB:49:50:00:18 -cn el2bcn16
# Call Script on el2bcn17
ssh -l root el2bcn17 /tmp/GenScripts/ -ip -nm -gw -m4 A0:C9:09:50:00:69 -m5 A0:C9:0A:50:00:69 -cn el2bcn17
# Call Script on el2bcn18
ssh -l root el2bcn18 /tmp/GenScripts/ -ip -nm -gw -m4 A0:FB:55:50:00:20 -m5 A0:FB:56:50:00:20 -cn el2bcn18
# Call Script on el2bcn19
ssh -l root el2bcn19 /tmp/GenScripts/ -ip -nm -gw -m4 A0:0B:FD:50:00:21 -m5 A0:0B:FE:50:00:21 -cn el2bcn19
# Call Script on el2bcn20
ssh -l root el2bcn20 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D8:2D:50:00:22 -m5 A0:D8:2E:50:00:22 -cn el2bcn20
# Call Script on el2bcn21
ssh -l root el2bcn21 /tmp/GenScripts/ -ip -nm -gw -m4 A0:C3:65:50:00:23 -m5 A0:C3:66:50:00:23 -cn el2bcn21
# Call Script on el2bcn22
ssh -l root el2bcn22 /tmp/GenScripts/ -ip -nm -gw -m4 A0:DA:29:50:00:24 -m5 A0:DA:2A:50:00:24 -cn el2bcn22
# Call Script on el2bcn23
ssh -l root el2bcn23 /tmp/GenScripts/ -ip -nm -gw -m4 A0:1A:99:50:00:25 -m5 A0:1A:9A:50:00:25 -cn el2bcn23
# Call Script on el2bcn24
ssh -l root el2bcn24 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D7:6E:50:00:26 -m5 A0:D7:6D:50:00:26 -cn el2bcn24
# Call Script on el2bcn25
ssh -l root el2bcn25 /tmp/GenScripts/ -ip -nm -gw -m4 A0:D9:EA:50:00:27 -m5 A0:D9:E9:50:00:27 -cn el2bcn25
# Call Script on el2bcn26
ssh -l root el2bcn26 /tmp/GenScripts/ -ip -nm -gw -m4 A0:F9:8E:50:00:28 -m5 A0:F9:8D:50:00:28 -cn el2bcn26
# Call Script on el2bcn27
ssh -l root el2bcn27 /tmp/GenScripts/ -ip -nm -gw -m4 A0:F9:D6:50:00:29 -m5 A0:F9:D5:50:00:29 -cn el2bcn27
# Call Script on el2bcn28
ssh -l root el2bcn28 /tmp/GenScripts/ -ip -nm -gw -m4 A0:F9:22:50:00:30 -m5 A0:F9:21:50:00:30 -cn el2bcn28
# Call Script on el2bcn29
ssh -l root el2bcn29 /tmp/GenScripts/ -ip -nm -gw -m4 A0:FC:02:50:00:31 -m5 A0:FC:01:50:00:31 -cn el2bcn29
# Call Script on el2bcn30
ssh -l root el2bcn30 /tmp/GenScripts/ -ip -nm -gw -m4 A0:F9:1E:50:00:32 -m5 A0:F9:1D:50:00:32 -cn el2bcn30
/opt/ -g nodelist rm -r /tmp/GenScripts


./ -d <dcli home> -n <nodelist file> -ip <File containing IP Addresses> -gw <bond1 Gateway> -nm <bond1 Netmask> -vnic1 <Output of showvnics from switch 1> -vnic2 <Output of showvnics from switch 2> [-vnic3 <Output of showvnics from switch 3> -vnic4 <Output of showvnics from switch 4>]


-n : A file containing a list of nodes (one per line) that will be processes.
-ip : A file containing the IP Addresses to be used for each of the nodes. The file must be configured to have one IP Address per line.
-gw : Gateway to be used for the bond1.
-nm : Netmask to be used for bond1.
-vnic1 : File containing the output for showvnics on Gateway Switch 1. The contents will be paired with the entry for -vnics2 to create the ifcfg-eth4 file. (Compute Nodes 1-16)
-vnic2 : File containing the output for showvnics on Gateway Switch 2.The contents will be paired with the entry for –vnics1 to create the ifcfg-eth5 file. (Compute Nodes 1-16)
-vnic3 : File containing the output for showvnics on Gateway Switch 3. The contents will be paired with the entry for –vnics4 to create the ifcfg-eth4 file. (Compute Nodes 17-30)
-vnic4 : File containing the output for showvnics on Gateway Switch 4. The contents will be paired with the entry for –vnics3 to create the ifcfg-eth5 file. (Compute Nodes 17-30)

This is a simple utility script that will allow the user to create a number of VLANs for a specified IP range on a compute node bond. It assumes that the range of IP Addresses will be consecutive and all use the same NetMask.


./ [-ip Starting IP] [-nm Netmask] [-c Number of IPs require] [-b Bond]


-ip : Starting IP Address.
-nm: NetMask to be used
-c: Number of IP Addresses to create
-b: Bond to which the IP Addresses should be attached

As part of the EIS check list document the engineer is required to modify the /etc/host file to replace the localdomain with the domain name and edit the /etc/sysconfig/network file to, potentially, remove the NISDOMAIN entry and make the HOSTNAME fully qualified. This script will implement those changes and should be executed using the dcli command below:
/opt/ -g nodelist –x
On completion of the script the existing files will be backup in their source directory and modified as follows:





Before el2bcn01 el2bcn01-priv.localdomain el2bcn01-priv el2bcn02 el2bcn02-priv.localdomain el2bcn02-priv el2bcn03 el2bcn03-priv.localdomain el2bcn03-priv el2bcn04 el2bcn04-priv.localdomain el2bcn04-priv

After el2bcn01 el2bcn01-priv el2bcn02 el2bcn02-priv el2bcn03 el2bcn03-priv el2bcn04 el2bcn04-priv


./ [-k <Flag to indicate that the NISDOMAIN Entry should be kept>]


-k : A flag to indicate if the NISDOMAIN entry must be kept if not set it assume that the entry will be commented out.

Software Configuration Scripts

This script will be used to setup a default structure for a simple Weblogic installation on a configured rack allowing for a shared installation and creation of multiple node managers that will be stored in a common shared location. It does not attempt to install WLS (because there are currently 3 alternative solutions for this) but assumes that the user will do that later. It will simply prepare each compute node.
It assumes that the following shares have been created on the ZFS storage array.

  1. /export/ACSExalogicSystem/domains
  2. /export/ACSExalogicSystem/logs
  3. /export/ACSExalogicSystem/stores
  4. /export/ACSExalogicSystem/nodemgrs
  5. /export/products/Middleware11gPS3
The steps taken to prepare the system are as follows:

Recreate Trust Relation

This is an optional section of the script and requires that the user passes in the –e flag. It should be execute to re-enable the ssh trust between the nodes that was removed at the end of the “Exalogic Configuration Utility”.

Create oracle Group

Here we add the oracle group to each compute node specifying the group id=500.
groupadd -g 500 oracle

Create Users

By default we will create 3 users on each of the compute nodes. These users are oracle, weblogic and coherence. Once created the will all be given the password defined in the script variable $PASSWORD (see usage).
useradd -g oracle -u 500 oracle
useradd -g oracle -u 501 weblogic
useradd -g oracle -u 502 coherence

Create Local Directories

On each compute node we will create 7 local directories and change them so that they are owned by oracle:oracle. The directories created are :

  1. /u01/common/general
  2. /u01/common/patches
  3. /u01/products/Middleware11gPS3
  4. ($ACSWLSDOMAINSLOCAL)=/u01/ACSExalogicSystem/domains
  5. ($ACSWLSLOGSLOCAL)=/u01/ACSExalogicSystem/logs
  6. ($ACSWLSSTORESLOCAL)=/u01/ACSExalogicSystem/stores
  7. ($ACSWLSNODEMGRSLOCAL)=/u01/ACSExalogicSystem/nodemgrs

Modify /etc/fstab

To allow mounting of these directories to the appropriate ZFS project the /etc/fstab file is modified to add new mount commands for each of the local directories / mount point combinations.

Mount File Systems

Once the /etc/fstab has been modified all the directories are mounted. On completion of this the user can then chose one compute node to install WLS and configure using one of the multiple installation scripts.


./ [-s <Storage Node hostname>] [-d <dcli home>] [-n <nodelist file>] [-p <new user password>] [-e re-establish ssh trust]


-k : Name of the ZFS storage node so that it can be accessed from this machine.
-d : Home directory of the dcli command.
-n : List of nodes to configured (used by dcli).
-p : Password to be used for the oracle, weblogic and coherence users. The default is welcome1.
-e : Flag to indicate if the ssh trust must be re-established.

No comments:

Post a Comment