Wednesday, 10 August 2011

Exalogic Networking Part 1

This is the first blog entry in a series I will be writing on Exalogic Networking. The entries will start with the low level basics and work their way up to architectural strategies for the partitioning and configuration an Exalogic Rack.

This first entry will look at the low level creation and configuration of vNICs, vIPs, vLANs, BONDs and how we can leverage scripting and the dcli command to configure all Exalogic Compute Nodes at the same time. At this point in time the scripts with documented within this page are designed from Oracle Enterprise Linux but can be tweaked slightly for a Solaris Exalogic installation. In addition we will focus on the external facing 10Gb network because the the 1Gb is management only and should be on a private network whilst the internal InfiniBand network will not be directly accessible to the outside world.


vNICs

Virtual Network Interface Cards (vNICs) emulates a NIC for a given Compute Node within the Exalogic Rack. As part of the installation and configuration the Oracle Engineer will create a number of these, based on a predefined algorithm, to allow 10Gb connection through the  switches within the rack. Internally this is implemented over the InfiniBand Network and converted to 10Gb at the switch.

As mentioned the Engineer will create the vNICs with MAC addresses based on a predefined algorithm that takes the GUID for each Compute Nodes Network card and a value indicating which exernal Ethernet port we will be connecting it to. In reality the only requirement is that the MAC addresses be unique. Given the fact that hand cranking these for a full, 30 node, rack is time consuming and error prone I created a number of scripts that generate a simple script file containing all the required createvnic commands.

#!/bin/sh
# "el2bcn15 192.168.10.67 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:f5:fa -mac a0:f5:fa:10:00:67 -pkey default
# "el2bcn16 192.168.10.68 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:fb:4a -mac a0:fb:4a:10:00:68 -pkey default
# "el2bcn08 192.168.10.58 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d5:85 -mac a0:d5:85:10:00:58 -pkey default
# "el2bcn09 192.168.10.59 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d6:fe -mac a0:d6:fe:10:00:59 -pkey default
# "el2bcn12 192.168.10.62 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d5:6e -mac a0:d5:6e:10:00:62 -pkey default
# "el2bcn11 192.168.10.61 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d7:ea -mac a0:d7:ea:10:00:61 -pkey default
# "el2bcn14 192.168.10.64 VNIC
createvnic 0A-ETH-1 -guid 00:21:28:00:01:a1:0c:0e -mac a0:0c:0e:10:00:64 -pkey default
# "el2bcn13 192.168.10.63 VNIC
Once generated this file can be executed on the switches to create the actual vNICs that can then be used by the compute node to access / be accessed from external systems using the 10Gb network. Before this can be done we will need to create / configure the Ethernet  and Bonds on each compute node as described in the next section.

The two scripts below, genVNICsFromLinkListUp.sh and genVNICsFromIBNetDiscover.sh, use the existing Exalogic scripts linklistup and inbnetdiscover to obtain information about the Exalogic network configuration. If you already have a 10Gb cable plugged into the Exalogic switch then using the genVNICsFromLinkListUp.sh will identify this and create the script file based on the active link by calling the genVNICsFromIBNetDiscover.sh passing the information. If on the other hand you need to create the vNICs before the cable has been connected then you can use the genVNICsFromIBNetDiscover.sh directly passing the Ethernet port id that will be used.

genVNICsFromLinkListUp.sh

  1 #!/bin/sh
  2 
  3 ################################################################################
  4 #
  5 #       Exalogic EL X2-2 1.0 (Linux x86-64) Configuration Script.
  6 #
  7 #  HEADER START
  8 # 
  9 #  THIS SCRIPT IS PROVIDED ON AN �AS IS� BASIS, WITHOUT WARRANTY OF ANY KIND, 
 10 #  EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT 
 11 #  THE COVERED SCRIPT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR 
 12 #  PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE 
 13 #  OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE 
 14 #  DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER 
 15 #  CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION.
 16 #  NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 
 17 #  DISCLAIMER.
 18 #
 19 #  When distributing this Code, include this HEADER in each file.
 20 #  If applicable, add the following below this this HEADER, with the fields
 21 #  enclosed by brackets "[]" replaced with your own identifying information:
 22 #       Portions Copyright [yyyy] [name of copyright owner]
 23 # 
 24 #  HEADER END
 25 # 
 26 # 
 27 #       Copyright 2011 Andrew Hopkinson, Oracle Corporation UK Ltd.
 28 #
 29 ################################################################################
 30 
 31 
 32 ################################################################################
 33 #
 34 # genVNICsFromListLinkUp.sh
 35 # =========================
 36 #
 37 # Version : 1.0.1
 38 #
 39 # This script does not directly affect the system it is running rather it calls
 40 # the genVNICsFromIBNetDiscover.sh script for each ETH port it finds which is up.
 41 # On execution of the genVNICsFromIBNetDiscover.sh script it will pass the ETH
 42 # port and output from the ibnetdiscover.
 43 #
 44 # Parameters
 45 #
 46 #   -f : Specifies a file that contain the output from the listlinkup command
 47 #      : executed on the switch. If no file is provided the script will execute
 48 #      : the listlinkup command to generate the file.
 49 #   -i : If this script is not being run on a switch or you already have an
 50 #      : output from ibnetdiscover that you want to use then it can be specified
 51 #      : with this flag. If the flag is not present the script will execute 
 52 #      : ibnetdiscover to generate the file.
 53 #   -v : If the script is not being run on a switch then you must supply the 
 54 #      : output from showvnics. If the flag is not present then the script will
 55 #      : execute showvnics and write the content to a temporary file.
 56 #
 57 # Version History
 58 # 1.0.0 - Initial Realease
 59 # 1.0.1 - Modified the generation of the delete VNICs script so that it is 
 60 #         generated based on the output from showvnics
 61 #
 62 ################################################################################
 63 
 64 
 65 IBNETDISCOVER=
 66 LISTLINKUPOUT=
 67 SHOWVNICSFILE=
 68 
 69 while [ $# -gt 0 ]
 70 do
 71         case "$1" in
 72                 -f) LISTLINKUPOUT="$2"; shift;;
 73                 -i) IBNETDISCOVER="$2"; shift;;
 74                 -v) SHOWVNICSFILE="$2"; shift;;
 75                 *) echo ""; echo >&2 \
 76                     "usage: $0 [-f listlinkup output] [-i ibnetdiscover output] [-v showvnics output] "
 77                     echo""; exit 1;;
 78                 *) break;;
 79         esac
 80         shift
 81 done
 82 
 83 # Generate ibnetdiscover
 84 if test "$IBNETDISCOVER" = ""
 85 then
 86         echo "Generating ibnetdiscover"
 87         echo ""
 88         IBNETDISCOVER=ibnetdiscover.out
 89         ibnetdiscover > $IBNETDISCOVER
 90         if [ "$?" -ne "0" ]
 91         then
 92                 echo ""
 93                 echo "Error: Generating ibnetdiscover output only works on the switch"
 94                 echo ""
 95                 exit 1
 96         fi
 97 fi
 98 
 99 # Generate showvnics
100 if test "$SHOWVNICSFILE" = ""
101 then
102         echo "Generating showvnics"
103         echo ""
104         SHOWVNICSFILE=showvnics.out
105         showvnics > $SHOWVNICSFILE
106         if [ "$?" -ne "0" ]
107         then
108                 echo ""
109                 echo "Error: Generating showvnics output only works on the switch"
110                 echo ""
111                 exit 1
112         fi
113 fi
114 
115 # Generate listlinkup
116 if test "$LISTLINKUPOUT" = ""
117 then
118         echo "Generating listlinkup"
119         echo ""
120         LISTLINKUPOUT=listlinkup.out
121         listlinkup > $LISTLINKUPOUT
122         if [ "$?" -ne "0" ]
123         then
124                 echo ""
125                 echo "Error: Generating listlinkup output only works on the switch"
126                 echo ""
127                 exit 1
128         fi
129 fi
130 
131 TEMPBRIDGEFILE=.upEthBridges
132 
133 grep Bridge $LISTLINKUPOUT | grep "ETH-" | grep "up" > $TEMPBRIDGEFILE
134 
135 while read line
136 do
137         i=0
138         for el in $line
139         do
140                 i=`expr $i + 1`
141 #               echo "$i = $el"
142                 if [ $i -eq 3 ]
143                 then
144 #                       echo "Port $el"
145                         ./genVNICsFromIBNetDiscover.sh -f $IBNETDISCOVER -p $el -v $SHOWVNICSFILE
146                         break
147                 fi
148         done
149 done < $TEMPBRIDGEFILE
150 
151 # Cleanup
152 
153 rm $TEMPBRIDGEFILE
154 
155 

genVNICsFromIBNetDiscover.sh

  1 #!/bin/sh
  2 
  3 ################################################################################
  4 #
  5 #       Exalogic EL X2-2 1.0 (Linux x86-64) Configuration Script.
  6 #
  7 #  HEADER START
  8 # 
  9 #  THIS SCRIPT IS PROVIDED ON AN �AS IS� BASIS, WITHOUT WARRANTY OF ANY KIND, 
 10 #  EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT 
 11 #  THE COVERED SCRIPT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR 
 12 #  PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE 
 13 #  OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE 
 14 #  DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER 
 15 #  CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION.
 16 #  NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 
 17 #  DISCLAIMER.
 18 #
 19 #  When distributing this Code, include this HEADER in each file.
 20 #  If applicable, add the following below this this HEADER, with the fields
 21 #  enclosed by brackets "[]" replaced with your own identifying information:
 22 #       Portions Copyright [yyyy] [name of copyright owner]
 23 # 
 24 #  HEADER END
 25 # 
 26 # 
 27 #       Copyright 2011 Andrew Hopkinson, Oracle Corporation UK Ltd.
 28 #
 29 ################################################################################
 30 
 31 
 32 ################################################################################
 33 #
 34 # genVNICsFromIBNetDiscover.sh
 35 # ============================
 36 #
 37 # Version : 1.0.2
 38 #
 39 # This does not affect the system directly rather it build a number of other 
 40 # scripts that the user can execute to modify the system. The scripts it 
 41 # created are in in the format :
 42 #      create---VNICs.sh
 43 #
 44 # This file contain a series of createvnic commands similar to those below
 45 #
 46 #       #!/bin/sh
 47 #       # "el01cn15 192.168.10.17 VNIC
 48 #       createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:f5:fa -mac a0:f5:fa:10:00:17 -pkey default
 49 #       # "el01cn16 192.168.10.18 VNIC
 50 #       createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:fb:4a -mac a0:fb:4a:10:00:18 -pkey default
 51 #       # "el01cn08 192.168.10.8 VNIC
 52 #       createvnic 0A-ETH-1 -guid 00:21:28:00:01:a0:d5:85 -mac a0:d5:85:10:00:08 -pkey default
 53 #
 54 # Once created the user can then execute this file on the switch or edit it as
 55 # required.
 56 #
 57 # Parameters
 58 #
 59 #   -a : Specifies is the script is to generate create-$swname-$ETHPORT-VNICs.sh
 60 #      : files for all switches that can be identified from the ibnetdiscover
 61 #      : output. By default this is false and this script will only generate
 62 #      : output for the first switch found that is the one it is being run on.
 63 #   -f : If this script is not being run on a switch or you already have an
 64 #      : output from ibnetdiscover that you want to use then it can be specified
 65 #      : with this flag. If the flag is not present the script will execute 
 66 #      : ibnetdiscover to generate the file.
 67 #   -p : Defines the ETH Port for which thes create..... script will be generated
 68 #      : by default we assume thise is 0A-ETH-1. 
 69 #   -v : If the script is not being run on a switch then you must supply the 
 70 #      : output from showvnics. If the flag is not present then the script will
 71 #      : execute showvnics and write the content to a temporary file.
 72 #   -c : Flag to indicate that the /conf/bx.conf should be cleared.
 73 #
 74 # When called from the genVNICsFromListLookUp.sh script -a will be ommited 
 75 # but -f and -p will be specified.
 76 #
 77 # Version History
 78 # 1.0.0 - Initial Realease
 79 # 1.0.1 - Modified the generation of the delete VNICs script so that it is 
 80 #         generated based on the output from showvnics
 81 # 1.0.2 - Added " service bxm restart" to the end of the deletevnincs generated 
 82 #         file.
 83 #
 84 ################################################################################
 85 
 86 
 87 IBNETDISCOVER=
 88 ETHPORT=0A-ETH-1
 89 PORTNO=00
 90 ALLSWITCHES=false
 91 SHOWVNICSFILE=
 92 CLEARBXCONF=false
 93 
 94 while [ $# -gt 0 ]
 95 do
 96         case "$1" in
 97                 -a) ALLSWITCHES=true;;
 98                 -c) CLEARBXCONF=true;;
 99                 -f) IBNETDISCOVER="$2"; shift;;
100                 -p) ETHPORT="$2"; shift;;
101                 -v) SHOWVNICSFILE="$2"; shift;;
102                 *) echo ""; echo >&2 \
103                     "usage: $0 [-f ibnetdiscover output] [-p Ethernet Port (eg 0A-ETH-1)] [-a generate for all visible switches] [-v showvnics output] [-c clear /conf/bx.conf]"
104                     echo""; exit 1;;
105                 *) break;;
106         esac
107         shift
108 done
109 #                   "usage: $0 [-f ibnetdiscover output] [-p Ethernet Port (eg 0A-ETH-1)] [-n Machine Name Prefix (eg el01)]"
110 
111 # Generate ibnetdiscover
112 if test "$IBNETDISCOVER" = ""
113 then
114         echo "Generating ibnetdiscover"
115         echo ""
116         IBNETDISCOVER=ibnetdiscover.out
117         ibnetdiscover > $IBNETDISCOVER
118         if [ "$?" -ne "0" ]
119         then
120                 echo ""
121                 echo "Error: Generating ibnetdiscover output only works on the switch"
122                 echo ""
123                 exit 1
124         fi
125 fi
126 
127 # Generate showvnics
128 if test "$SHOWVNICSFILE" = ""
129 then
130         echo "Generating showvnics"
131         echo ""
132         SHOWVNICSFILE=showvnics.out
133         showvnics > $SHOWVNICSFILE
134         if [ "$?" -ne "0" ]
135         then
136                 echo ""
137                 echo "Error: Generating showvnics output only works on the switch"
138                 echo ""
139                 exit 1
140         fi
141 fi
142 
143 
144 if test "$ETHPORT" = "0A-ETH-1"
145 then
146         PORTNO=10
147 fi
148 if test "$ETHPORT" = "0A-ETH-2"
149 then
150         PORTNO=20
151 fi
152 if test "$ETHPORT" = "0A-ETH-3"
153 then
154         PORTNO=30
155 fi
156 if test "$ETHPORT" = "0A-ETH-4"
157 then
158         PORTNO=40
159 fi
160 if test "$ETHPORT" = "1A-ETH-1"
161 then
162         PORTNO=50
163 fi
164 if test "$ETHPORT" = "1A-ETH-2"
165 then
166         PORTNO=60
167 fi
168 if test "$ETHPORT" = "1A-ETH-3"
169 then
170         PORTNO=70
171 fi
172 if test "$ETHPORT" = "1A-ETH-4"
173 then
174         PORTNO=80
175 fi
176 
177 DELETEVNICSFILE=
178 CREATEVNICSFILE=
179 
180 # Process file
181 
182 switchcount=0
183 ln=0
184 vn=0
185 while read line
186 do
187         ln=`expr $ln + 1`
188         # Check for IB Switch
189         i=0
190         createfiles=false
191         swname=Unknown
192         for el in $line
193         do
194                 i=`expr $i + 1`
195                 if [ $i -eq 1 ]
196                 then
197                         if test "$el" != "Switch"
198                         then
199                                 break
200                         fi
201                 fi
202                 if [ $i -eq 6 ]
203                 then
204                         if test "$el" != "IB"
205                         then
206                                 break
207                         fi
208                 fi
209                 if [ $i -eq 7 ]
210                 then
211                         if test "$el" != "QDR"
212                         then
213                                 break
214                         fi
215                 fi
216                 if [ $i -eq 10 ]
217                 then
218                         switchname=$el
219                         if test "$el" != "localhost"
220                         then
221                                 swname=$el
222                                 createfiles=true
223                                 break
224                         fi
225                 fi
226                 if [ $i -eq 11 ]
227                 then
228                         swname=$el
229                         createfiles=true
230                         break
231                 fi
232 #               echo "$i el = $el"
233         done
234         
235         # Check if we need to create files
236         if test "$createfiles" = "true"
237         then
238                 createfiles=false
239                 switchcount=`expr $switchcount + 1`
240                         
241                 if [ $switchcount -eq 2 ]
242                 then
243                         if test "$ALLSWITCHES" != "true"
244                         then
245                                 break
246                         fi
247                 fi
248                 
249                 CREATEVNICSFILE=create-$swname-$ETHPORT-VNICs.sh
250                 echo "#!/bin/sh" > $CREATEVNICSFILE
251                 chmod +x $CREATEVNICSFILE
252         fi
253         
254         # Check for Connection
255         i=0
256         for el in $line
257         do
258                 i=`expr $i + 1`
259                 if [ $i -eq 2 ]
260                 then
261                         sc=`expr substr $el 2 1`
262                         if test "$sc" != "H"
263                         then
264                                 break
265                         fi
266                         
267                         oct1=`expr substr $el 25 2`
268                         oct2=`expr substr $el 27 2`
269                         oct3=`expr substr $el 29 2`
270                         oct4=`expr substr $el 31 2`
271                         oct5=`expr substr $el 33 2`
272                         oct6=`expr substr $el 35 2`
273                         oct7=`expr substr $el 37 2`
274 
275                         guid=00:$oct1:$oct2:$oct3:$oct4:$oct5:$oct6:$oct7
276                 fi
277                 if [ $i -eq 4 ]
278                 then
279                         cn=$el
280                 fi
281                 if [ $i -eq 5 ]
282                 then
283                         if test "$el" != "EL-C"
284                         then
285                                 break
286                         fi
287                 fi
288                 if [ $i -eq 6 ]
289                 then
290                         ip=$el
291                         len=`expr length $el`
292                         start=`expr $len - 1`
293                         mac3=`expr substr $el $start 2`
294                         if test "`expr substr $mac3 1 1`" = "."
295                         then
296                                 start=`expr $start + 1`
297                                 mac3=0`expr substr $el $start 1`
298                         fi
299 
300                         mac=a0:$oct6:$oct7:$PORTNO:00:$mac3
301                         
302                         # Write to file
303                         echo "# $cn $ip VNIC"  >> $CREATEVNICSFILE
304                         echo "createvnic $ETHPORT -guid $guid -mac $mac -pkey default" >> $CREATEVNICSFILE
305                         
306                 fi
307 #               echo "$i el = $el"
308         done
309         
310 done < $IBNETDISCOVER
311 
312 
313 echo "Generated Switch Files"
314 
315 # Generate DELETEVNICS File
316 DELETEVNICSFILE=delete-$ETHPORT-VNICs.sh
317 echo "#!/bin/sh" > $DELETEVNICSFILE
318 chmod +x $DELETEVNICSFILE
319 
320 SHOWVNICSFILEGREP=$SHOWVNICSFILE.grep
321 grep $ETHPORT $SHOWVNICSFILE > $SHOWVNICSFILEGREP
322 while read line
323 do
324         i=0
325         for el in $line
326         do
327                 i=`expr $i + 1`
328                 if [ $i -eq 1 ]
329                 then
330                         vn=$el
331                         echo "deletevnic $ETHPORT $vn" >> $DELETEVNICSFILE
332                         break
333                 fi
334         done
335 done < $SHOWVNICSFILEGREP
336 
337 rm $SHOWVNICSFILEGREP
338 
339 if test "$CLEARBXCONF" = "true"
340 then
341         echo "echo > /conf/bx.conf" >> $DELETEVNICSFILE
342 fi
343 
344 echo "service bxm restart" >> $DELETEVNICSFILE
345 
346 echo "Generated Delete VNICS Files"
347 
348 

Bonds and Eth Config

Once the vNICs have been created we need to configure each of the Compute Nodes to connect to each of the switches and to do this we will create two Ethernet configuration files, by convention named ifcfg-eth4 and ifcfg-eth5, which are the physical connections to the ports. In addition we will create a bonded port, by convention name ifcfg-bond1, which it the logical port that the Compute Node will connect through. The creation of the bonded (bond1) network provides network resilience and fail-over. The contents of these files are similar to the following :

Ifcfg-eth4

DEVICE=eth4
BOOTPROTO=none
ONBOOT=yes
HWADDR=A0:F9:1E:50:00:32
MASTER=bond1
SLAVE=yes

Ifcfg-eth5

DEVICE=eth5
BOOTPROTO=none
ONBOOT=yes
HWADDR=A0:F9:1D:50:00:32
MASTER=bond1
SLAVE=yes

Ifcfg-bond1

DEVICE=bond1
IPADDR=10.131.12.94
NETMASK=255.255.255.0
BOOTPROTO=none
USERCTL=no
TYPE=Ethernet
ONBOOT=yes
IPV6INIT=no
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"
GATEWAY=10.131.12.1
In addition to allow default routing over the 10Gb Network we will need to modify the network conguration to specify the 10Gb network as the default routing option.

Again I have created some simple scripts to simplify the work and running the genCNNetFiles.sh will generated an new script that uses dcli to distribute the genIfcfgFiles.sh and execute it using ssh. This will generate the appropriate ifcfg files based on the outputs from the previous create vNICs execution.

genCNNetFiles.sh

  1 #!/bin/sh
  2 
  3 ################################################################################
  4 #
  5 #       Exalogic EL X2-2 1.0 (Linux x86-64) Configuration Script.
  6 #
  7 #  HEADER START
  8 # 
  9 #  THIS SCRIPT IS PROVIDED ON AN �AS IS� BASIS, WITHOUT WARRANTY OF ANY KIND, 
 10 #  EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT 
 11 #  THE COVERED SCRIPT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR 
 12 #  PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE 
 13 #  OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE 
 14 #  DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER 
 15 #  CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION.
 16 #  NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 
 17 #  DISCLAIMER.
 18 #
 19 #  When distributing this Code, include this HEADER in each file.
 20 #  If applicable, add the following below this this HEADER, with the fields
 21 #  enclosed by brackets "[]" replaced with your own identifying information:
 22 #       Portions Copyright [yyyy] [name of copyright owner]
 23 # 
 24 #  HEADER END
 25 # 
 26 # 
 27 #       Copyright 2011 Andrew Hopkinson, Oracle Corporation UK Ltd.
 28 #
 29 ################################################################################
 30 
 31 
 32 ################################################################################
 33 #
 34 # genCNNetFiles.sh
 35 # ================
 36 #
 37 # Version : 1.0.2
 38 #
 39 # This does not affect the system directly rather it build a number of other 
 40 # scripts that the user can execute to modify the system. The three scripts it 
 41 # creates are:
 42 #    createAllIfCfg.sh
 43 #    setDefaultRouting.sh
 44 #    add10gIPToHosts.sh
 45 # In addition it relies on the existance of the genIfcfgFiles.sh within the same
 46 # directory.
 47 #
 48 # Parameters
 49 #
 50 #   -d     : Home directory of the dcli command
 51 #   -n     : Node list to need processed
 52 #   -ip    : A file containing a list (one per line) IP Address that will be
 53 #          : allocated to the nodes. This allows for non sequential IPs
 54 #   -nm    : The Bond1 Network Mask
 55 #   -gw    : The Gateway to be used by the 10 Gb bond1 conection.
 56 #   -vnic1 : File containing the output for showvnics on Gateway Switch 1 
 57 #          : The contents will be paired with the entry for -vnics2 to
 58 #          : create the ifcfg-eth4 file. (Compute Nodes 1-16)
 59 #   -vnic2 : File containing the output for showvnics on Gateway Switch 2 
 60 #          : The contents will be paired with the entry for -vnics1 to
 61 #          : create the ifcfg-eth5 file. (Compute Nodes 1-16)
 62 #   -vnic3 : File containing the output for showvnics on Gateway Switch 3 
 63 #          : The contents will be paired with the entry for -vnics4 to
 64 #          : create the ifcfg-eth4 file. (Compute Nodes 17-30)
 65 #   -vnic4 : File containing the output for showvnics on Gateway Switch 4 
 66 #          : The contents will be paired with the entry for -vnics3 to
 67 #          : create the ifcfg-eth5 file. (Compute Nodes 17-30)
 68 #
 69 # createAllIfCfg.sh
 70 # -----------------
 71 #
 72 # This generated script will use dcli to copy the genIfcfgFiles.sh to all nodes
 73 # and then execute this script using a series of ssh commands. These ssh commands
 74 # commands are built within the main script and pass in compute node specific 
 75 # information (see genIfcfgFiles.sh header for information). On completion of 
 76 # the script the genIfcfgFiles.sh will be removed from all compute nodes.
 77 #
 78 # setDefaultRouting.sh
 79 # --------------------
 80 #
 81 # This generated script will remove the default routing information and then 
 82 # add the 10 Gb (bond1) gateway as the default route. It should be executed 
 83 # after the service network restart has completed.
 84 #
 85 # add10gIPToHosts.sh
 86 # ------------------
 87 #
 88 # Optional script that when executed will add entries to the hosts file in the
 89 # format :
 90 #     <10Gb Suffix>
 91 #    10.131.12.61 el2bcn01-10g
 92 # The script is simply a set of echo commands to the /etc/hosts file and it is
 93 # left to the user to decide if this should be executed via dcli.
 94 #
 95 # Version History
 96 # 1.0.0 - Initial Realease.
 97 # 1.0.1 - Enable generation of add10gIPToHosts.sh script.
 98 # 1.0.2 - Extend the generated createAllIfCfg.sh so that it will re-run the ssh
 99 #         trust setup removed at the end of the Exalogic Configuration Utility.
100 #         This can be disabled by using the -i flag on the createAllIfCfg.sh.
101 #
102 ################################################################################
103 
104 NODELIST=nodelist
105 DCLIHOME=/opt/exalogic.tools/tools
106 VNICS1FILE=
107 VNICS2FILE=
108 VNICS3FILE=
109 VNICS4FILE=
110 IPFILE=
111 TENGBSUFFIX=-10g
112 
113 bond1nm=255.255.255.0
114 bond1gw=10.131.12.1
115 
116 while [ $# -gt 0 ]
117 do
118         case "$1" in
119                 -d) DCLIHOME="$2"; shift;;
120                 -n) NODELIST="$2"; shift;;
121                 -ip) IPFILE="$2"; shift;;
122                 -nm) bond1nm="$2"; shift;;
123                 -gw) bond1gw="$2"; shift;;
124                 -vnic1) VNICS1FILE="$2"; shift;;
125                 -vnic2) VNICS2FILE="$2"; shift;;
126                 -vnic3) VNICS3FILE="$2"; shift;;
127                 -vnic4) VNICS4FILE="$2"; shift;;
128                 -10g) TENGBSUFFIX="$2"; shift;;
129                 *) echo ""; echo >&2 \
130                     "usage: $0 -d  -n  -ip  -gw  -nm  -vnic1  -vnic2  [-vnic3  -vnic4 ]"
131                     echo""; exit 1;;
132                 *) break;;
133         esac
134         shift
135 done
136 
137 if test "$DCLIHOME" = ""
138 then
139         DCLIHOME=/opt/exalogic.tools/tools
140 fi
141 
142 if test "$NODELIST" = ""
143 then
144         NODELIST=nodelist
145 fi
146 
147 if test "$TENGBSUFFIX" = ""
148 then
149         TENGBSUFFIX=-10g
150 fi
151 
152 vl1=${#VNICS1FILE}
153 vl2=${#VNICS2FILE}
154 vl3=${#VNICS3FILE}
155 vl4=${#VNICS4FILE}
156 
157 # Check for the VNIC Files
158 if [ $vl1 -eq 0 ] || [ $vl2 -eq 0 ]
159 then
160         echo "Outputs from showvnics for Switch 1 and 2 must be provided"
161         exit 1
162 fi
163 
164 if [ $vl3 -gt 0 ] && [ $vl4 -eq 0 ]
165 then
166         echo "Output from Switch 3 and 4 must be provided as a pair"
167         exit 1
168 fi
169 if [ $vl3 -eq 0 ] && [ $vl4 -gt 0 ]
170 then
171         echo "Output from Switch 3 and 4 must be provided as a pair"
172         exit 1
173 fi
174 
175 # Need to validate the vnic file pairing ie there are either 2 or 4 I assume they are paired correctly
176 
177 GENIFCFGSCRIPT=genIfcfgFiles.sh
178 CREATECFGFILE=createAllIfCfg.sh
179 SETDEFAULTROUTING=setDefaultRouting.sh
180 ADDTOHOSTS=add10gIPToHosts.sh
181 
182 echo "#!/bin/sh" > $SETDEFAULTROUTING
183 echo "echo \"**********************\"" >> $SETDEFAULTROUTING
184 echo "echo \"** Processing \`hostname\` \"" >> $SETDEFAULTROUTING
185 echo "echo \"**********************\"" >> $SETDEFAULTROUTING
186 echo "netstat -rn" >> $SETDEFAULTROUTING
187 echo "route delete default" >> $SETDEFAULTROUTING
188 echo "route add default gw $bond1gw bond1" >> $SETDEFAULTROUTING
189 echo "netstat -rn" >> $SETDEFAULTROUTING
190 chmod +x $SETDEFAULTROUTING
191 
192 
193 
194 # Copy Create Script to all nodes
195 DESTDIR=/tmp/GenScripts
196 
197 echo "#!/bin/sh" > $CREATECFGFILE
198 chmod +x $CREATECFGFILE
199 
200 # Set ssh Trust Setup
201 echo "SETUPSSHTRUST=true"  >> $CREATECFGFILE
202 echo ""  >> $CREATECFGFILE
203 
204 echo "while [ \$# -gt 0 ]"  >> $CREATECFGFILE
205 echo "do"  >> $CREATECFGFILE
206 echo "  case \"\$1\" in "  >> $CREATECFGFILE
207 echo "          -i) SETUPSSHTRUST=false;;"  >> $CREATECFGFILE
208 echo "          *) echo \"\"; echo >&2 \\"  >> $CREATECFGFILE
209 echo "              \"usage: \$0 [-i ]\""  >> $CREATECFGFILE
210 echo "              echo\"\"; exit 1;;"  >> $CREATECFGFILE
211 echo "          *) break;;"  >> $CREATECFGFILE
212 echo "  esac"  >> $CREATECFGFILE
213 echo "  shift"  >> $CREATECFGFILE
214 echo "done"  >> $CREATECFGFILE
215 echo ""  >> $CREATECFGFILE
216 
217 echo "if test \"\$SETUPSSHTRUST\" = \"true\""  >> $CREATECFGFILE
218 echo "then"  >> $CREATECFGFILE
219 echo "  echo \"Setting up SSH Trust\""  >> $CREATECFGFILE
220 echo "  echo \"\""  >> $CREATECFGFILE
221 echo "  ssh-keygen -t dsa"  >> $CREATECFGFILE
222 echo "  $DCLIHOME/dcli -k -g $NODELIST "  >> $CREATECFGFILE
223 echo "fi"  >> $CREATECFGFILE
224 echo ""  >> $CREATECFGFILE
225 
226 # Copy Files
227 echo "chmod +x genIfcfgFiles.sh" >> $CREATECFGFILE
228 echo "dos2unix genIfcfgFiles.sh" >> $CREATECFGFILE
229 echo "$DCLIHOME/dcli -g $NODELIST mkdir -p $DESTDIR" >> $CREATECFGFILE
230 echo "$DCLIHOME/dcli -g $NODELIST -d $DESTDIR -f genIfcfgFiles.sh" >> $CREATECFGFILE
231 echo "$DCLIHOME/dcli -g $NODELIST -d $DESTDIR -f $SETDEFAULTROUTING" >> $CREATECFGFILE
232 echo ""  >> $CREATECFGFILE
233 
234 # Now for each compute node we need to execute the appropriate command
235 NEWNODELIST=$NODELIST.out
236 > $NEWNODELIST
237 
238 echo "#!/bin/sh" > $ADDTOHOSTS
239 chmod +x $ADDTOHOSTS
240 echo "echo \"\" >> /etc/hosts" >> $ADDTOHOSTS
241 echo "echo \"# Auto Generated Exalogic IP / Hostname for 10Gb (bond1) Connections\" >> /etc/hosts" >> $ADDTOHOSTS
242 echo "echo \"\" >> /etc/hosts" >> $ADDTOHOSTS
243 
244 cnip=0
245 # We need to execute this for the Switch pairings
246 # VNICS1FILE & VNICS2FILE 
247 # Optionally
248 # VNICS3FILE & VNICS4FILE
249 for ETH4FILE in $VNICS1FILE $VNICS3FILE
250 do
251         # Get ETH5FILE
252         if test "$ETH4FILE" = ""
253         then
254                 break
255         else
256                 if test "$ETH4FILE" = "$VNICS1FILE"
257                 then
258                         ETH5FILE=$VNICS2FILE
259                 else
260                         ETH5FILE=$VNICS4FILE
261                 fi
262         fi
263         echo "Eth4 $ETH4FILE"
264         echo "Eth5 $ETH5FILE"
265         
266         # Sort Input File
267         SORTEDETH4FILE=$ETH4FILE.sorted
268         sort +4 -5 $ETH4FILE > $SORTEDETH4FILE
269         
270         ln=0
271         while read line
272         do
273                 ln=`expr $ln + 1`
274                 
275                 i=0
276                 for el in $line
277                 do
278                         i=`expr $i + 1`
279                         if [ $i -eq 1 ]
280                         then
281                                 s=$el
282                                 break;
283                         fi
284                 done
285                 
286                 if test "$s" = "---"
287                 then
288                         continue
289                 fi
290                 if test "$s" = "ID"
291                 then
292                         continue
293                 fi
294                 # It's a Compute node so lets process it
295                 
296                 cnip=`expr $cnip + 1`
297                 # Get the Compute Node and its Mac from the first VNIC File
298                 # We will assume this is eth4 Mac address
299                 i=0
300                 for el in $line
301                 do
302                         i=`expr $i + 1`
303                         if [ $i -eq 5 ]
304                         then
305                                 cn=$el
306                                 echo "$cn" >> $NEWNODELIST
307                         fi
308                         if [ $i -eq 7 ]
309                         then
310                                 ip=$el
311                         fi
312                         if [ $i -eq 9 ]
313                         then
314                                 eth4mac=$el
315                         fi
316                         if [ $i -eq 12 ]
317                         then
318                                 port=$el
319                         fi
320                 done
321                 
322                 # Now we have the information from the first VNICs file we will process
323                 # the second to get eth5 mac address.
324                 #line=`grep $cn $VNICS2FILE`
325                 line=`grep $cn $ETH5FILE`
326                 #echo $line
327                 i=0
328                 for el in $line
329                 do
330                         i=`expr $i + 1`
331                         if [ $i -eq 9 ]
332                         then
333                                 eth5mac=$el
334                         fi
335                 done
336                 
337                 # Now we need to get the Bond 1 (10 Gb) IP Address from the list in the supplied file
338                 i=0
339                 while read ipaddr
340                 do
341                         i=`expr $i + 1`
342                         if [ $i -eq $cnip ]
343                         then
344                                 bond1ip=$ipaddr
345                                 break
346                         fi
347                 done < $IPFILE
348                 echo "# Call Script on $cn" >> $CREATECFGFILE
349                 echo "ssh -l root $cn $DESTDIR/$GENIFCFGSCRIPT -ip $bond1ip -nm $bond1nm -gw $bond1gw -m4 $eth4mac -m5 $eth5mac -cn $cn" >> $CREATECFGFILE
350                 
351                 cnalt=
352                 cnl=${#cn}
353                 if [ $cnl -gt 2 ]
354                 then
355                         ipos=`expr $cnl - 2`
356                         cns=${cn:0:$ipos}
357                         cne=${cn:$ipos}
358                         cnalt=${cn/%$cne/x$cne}
359                 fi
360                 echo "echo \"$bond1ip $cn$TENGBSUFFIX $cnalt\" >> /etc/hosts" >> $ADDTOHOSTS
361         done < $SORTEDETH4FILE
362         #done < $VNICS1FILE
363 done
364 
365 # Remove working file
366 echo "$DCLIHOME/dcli -g $NODELIST rm -r $DESTDIR" >> $CREATECFGFILE
367 
368 # Report Complete
369 echo ""
370 echo "Generated $CREATECFGFILE"
371 echo "=================================="
372 echo ""
373 echo "File should be validated before executing"
374 echo ""
375 echo "The script has also create the $SETDEFAULTROUTING file that can be executed"
376 echo "to reconfigure the default routing once the $CREATECFGFILE has completed"
377 echo "its restart of the network"
378 echo ""
379 
380 

genIfgcfgFiles.sh

  1 #!/bin/sh
  2 
  3 ################################################################################
  4 #
  5 #       Exalogic EL X2-2 1.0 (Linux x86-64) Configuration Script.
  6 #
  7 #  HEADER START
  8 # 
  9 #  THIS SCRIPT IS PROVIDED ON AN �AS IS� BASIS, WITHOUT WARRANTY OF ANY KIND, 
 10 #  EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT 
 11 #  THE COVERED SCRIPT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR 
 12 #  PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE 
 13 #  OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE 
 14 #  DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER 
 15 #  CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION.
 16 #  NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 
 17 #  DISCLAIMER.
 18 #
 19 #  When distributing this Code, include this HEADER in each file.
 20 #  If applicable, add the following below this this HEADER, with the fields
 21 #  enclosed by brackets "[]" replaced with your own identifying information:
 22 #       Portions Copyright [yyyy] [name of copyright owner]
 23 # 
 24 #  HEADER END
 25 # 
 26 # 
 27 #       Copyright 2011 Andrew Hopkinson, Oracle Corporation UK Ltd.
 28 #
 29 ################################################################################
 30 
 31 
 32 ################################################################################
 33 #
 34 # genIfcfgFiles.sh
 35 # ================
 36 #
 37 # Version : 1.0.0
 38 #
 39 # This script is used to generate ifcfg_bond1, ifcfg_eth4 and ifcfg_eth5 files
 40 # in the /etc/sysconfig/network-scripts directory based on the passed parameters.
 41 #
 42 # The key reason for this script is to help configure the 10 Gb Networking on
 43 # bond1 but also configure the redundant eth4 & eth5 network connection to our
 44 # pairs of switches.
 45 #
 46 # Parameters
 47 #
 48 #   -ip : The IP Address that should be assigned to the bond1 connection
 49 #   -nm : The Bond1 Network Mask
 50 #   -gw : The Gateway to be used by the 10 Gb bond1 conection.
 51 #   -m4 : The MAC Address to be associated with Eth4. This may have been 
 52 #         previously configured using the genVNICsFromIBNetDiscover.sh or
 53 #         genVNICsFromListLinkUp.sh on the Switches.
 54 #   -m5 : The MAC Address to be associated with Eth5. This may have been 
 55 #         previously configured using the genVNICsFromIBNetDiscover.sh or
 56 #         genVNICsFromListLinkUp.sh on the Switches.
 57 #
 58 # Processing
 59 #
 60 # On execution the script will first edit the /etc/sysconfig/network file to add
 61 # the specified 10 Gb Gateway. Once this has been done it will create the three
 62 # ifcfg- files in /etc/sysconfig/network-scripts. 
 63 #
 64 # Having created the file the script will, optionally, then edit the /etc/sysctl.conf
 65 # file to set net.ipv4.conf.default.rp_filter = 2 which will provide a loose 
 66 # filtering on the network and hence allow the standard management connection to be 
 67 # discovered and connected to.
 68 #
 69 # The final part of the script will restart the network service to implement the
 70 # changes. On completion of this script we will need to remove the default route
 71 # and add a new default that goes via the bond1 gateway. This is done in the 
 72 # setDefaultRouting.sh that will be generated as part of the genCNNetFiles.sh.
 73 #
 74 # Version History
 75 # 1.0.0 - Initial Realease
 76 #
 77 ################################################################################
 78 
 79 
 80 # Define Variables
 81 BAK_EXT=`date +"%Y%m%d-%H%M%S"`
 82 BAK_PREFIX=Orig
 83 
 84 SYSCONFROOT=/etc/sysconfig
 85 IFCFGROOT=$SYSCONFROOT/network-scripts
 86 
 87 NETWORKFILE=$SYSCONFROOT/network
 88 BACKUPNETWORKFILE=$SYSCONFROOT/$BAK_PREFIX.network.$BAK_EXT
 89 
 90 SYSCTLFILE=/etc/sysctl.conf
 91 BACKUPSYSCTLFILE=/etc/$BAK_PREFIX.sysctl.conf.$BAK_EXT
 92 
 93 BOND1IP=
 94 BOND1NM=
 95 BOND1GW=
 96 ETH4MAC=
 97 ETH5MAC=
 98 
 99 COMPUTENODE=
100 
101 # Setting the following variable to true will cause the /etc/sysctl.conf file
102 # to be edited so that the net.ipv4.conf.default.rp_filter = 2
103 # This is not recommended for production.
104 CHANGESYSCTL=false
105 
106 # Read Parameters
107 while [ $# -gt 0 ]
108 do
109         case "$1" in
110                 -r) IFCFGROOT="$2"; shift;;
111                 -ip) BOND1IP="$2"; shift;;
112                 -nm) BOND1NM="$2"; shift;;
113                 -gw) BOND1GW="$2"; shift;;
114                 -m4) ETH4MAC="$2"; shift;;
115                 -m5) ETH5MAC="$2"; shift;;
116                 -cn) COMPUTENODE="$2"; shift;;
117                 *) echo ""; echo >&2 \
118                     "usage: $0 [-r ] -ip  -nm  -gw  -m4  -m5  "
119                     echo""; exit 1;;
120                 *) break;;
121         esac
122         shift
123 done
124 
125 echo ""
126 echo "***** Starting Processessing on Compute Node $COMPUTENODE *****"
127 echo ""
128 
129 # Set File Names
130 
131 IFCFG_BOND1=$IFCFGROOT/ifcfg-bond1
132 IFCFG_ETH4=$IFCFGROOT/ifcfg-eth4
133 IFCFG_ETH5=$IFCFGROOT/ifcfg-eth5
134 
135 BAK_IFCFG_BOND1=$IFCFGROOT/$BAK_PREFIX.ifcfg-bond1.$BAK_EXT
136 BAK_IFCFG_ETH4=$IFCFGROOT/$BAK_PREFIX.ifcfg-eth4.$BAK_EXT
137 BAK_IFCFG_ETH5=$IFCFGROOT/$BAK_PREFIX.ifcfg-eth5.$BAK_EXT
138 
139 # Backup any existing files
140 if [ -f $IFCFG_BOND1 ] 
141 then
142         mv $IFCFG_BOND1 $BAK_IFCFG_BOND1
143 fi
144 if [ -f $IFCFG_ETH4 ] 
145 then
146         mv $IFCFG_ETH4 $BAK_IFCFG_ETH4
147 fi
148 if [ -f $IFCFG_ETH5 ] 
149 then
150         mv $IFCFG_ETH5 $BAK_IFCFG_ETH5
151 fi
152 
153 # Generate Bond1 (10 Gb Ethernet File)
154 
155 > $IFCFG_BOND1
156 echo "DEVICE=bond1" >> $IFCFG_BOND1
157 echo "IPADDR=$BOND1IP" >> $IFCFG_BOND1
158 echo "NETMASK=$BOND1NM" >> $IFCFG_BOND1
159 echo "BOOTPROTO=none" >> $IFCFG_BOND1
160 echo "USERCTL=no" >> $IFCFG_BOND1
161 echo "TYPE=Ethernet" >> $IFCFG_BOND1
162 echo "ONBOOT=yes" >> $IFCFG_BOND1
163 echo "IPV6INIT=no" >> $IFCFG_BOND1
164 echo "BONDING_OPTS=\"mode=active-backup miimon=100 downdelay=5000 updelay=5000\"" >> $IFCFG_BOND1
165 # Do I need the Gateway or just set it in /etc/sysconfig/network as per EIS Checklist
166 echo "GATEWAY=$BOND1GW" >> $IFCFG_BOND1
167 
168 # Process Network file
169 # Backup
170 mv $NETWORKFILE $BACKUPNETWORKFILE
171 
172 while read line
173 do
174         len=${#line}
175         if [ $len -eq 0 ]
176         then
177                 continue
178         fi
179         f7=`expr substr $line 1 7`
180         if test "$f7" != "GATEWAY"
181         then
182                 echo $line >> $NETWORKFILE
183         fi
184 done < $BACKUPNETWORKFILE
185 echo "GATEWAY=$BOND1GW" >> $NETWORKFILE
186 echo "GATEWAYDEV=bond1" >> $NETWORKFILE
187 
188 echo "Edited $NETWORKFILE file"
189 
190 # Generate ETH4 
191 
192 > $IFCFG_ETH4
193 echo "DEVICE=eth4" >> $IFCFG_ETH4
194 echo "BOOTPROTO=none" >> $IFCFG_ETH4
195 echo "ONBOOT=yes" >> $IFCFG_ETH4
196 echo "HWADDR=$ETH4MAC" >> $IFCFG_ETH4
197 echo "MASTER=bond1" >> $IFCFG_ETH4
198 echo "SLAVE=yes" >> $IFCFG_ETH4
199 
200 # Generate ETH5 
201 
202 > $IFCFG_ETH5
203 echo "DEVICE=eth5" >> $IFCFG_ETH5
204 echo "BOOTPROTO=none" >> $IFCFG_ETH5
205 echo "ONBOOT=yes" >> $IFCFG_ETH5
206 echo "HWADDR=$ETH5MAC" >> $IFCFG_ETH5
207 echo "MASTER=bond1" >> $IFCFG_ETH5
208 echo "SLAVE=yes" >> $IFCFG_ETH5
209 
210 
211 # Complete Message
212 
213 echo "Generated New ifcfg files"
214 echo "          $IFCFG_BOND1"
215 echo "          $IFCFG_ETH4"
216 echo "          $IFCFG_ETH5"
217 
218 if test "$CHANGESYSCTL" = "true"
219 then
220         # Change the /etc/sysctl.conf so that the default filtering is 2
221         # This will allow the normal & 10 Gb network access
222 
223         # Backup
224         mv $SYSCTLFILE $BACKUPSYSCTLFILE
225         
226         while read line
227         do
228                 len=${#line}
229                 if [ $len -eq 0 ]
230                 then
231                         echo "" >> $SYSCTLFILE
232                 fi
233                 
234                 i=0
235                 for el in $line
236                 do
237                         i=`expr $i + 1`
238                         if [ $i -eq 1 ]
239                         then
240                                 if test "$el" = "net.ipv4.conf.default.rp_filter"
241                                 then
242                                         echo "net.ipv4.conf.default.rp_filter = 2" >> $SYSCTLFILE
243                                 else
244                                         echo $line >> $SYSCTLFILE
245                                 fi
246                                 break
247                         fi
248                 done
249         done < $BACKUPSYSCTLFILE
250         
251         echo "Editted $SYSCTLFILE file"
252 fi
253 
254 # Restarting Network
255 
256 service network restart
257 
258 echo ""
259 echo "***** Finished Processessing on Compute Node $COMPUTENODE *****"
260 echo ""
261 
262 

vIPs

Virtual IPs are useful when multiple, virtual, IPs for a single compute node and we have used this when running multiple WLS domains on the same Compute Node. Using vIPs allows you the create all the domains with the same port number but different IP address. The net result is that we can simplify HA / Failover and load balancing.

The simple script below will generate a specified number of, sequential, vIPs with a given Netmask for a given bond.

genVNICsFromLinkListUp.sh

  1 #!/bin/sh
  2 
  3 ################################################################################
  4 #
  5 #       Exalogic EL X2-2 1.0 (Linux x86-64) Configuration Script.
  6 #
  7 #  HEADER START
  8 # 
  9 #  THIS SCRIPT IS PROVIDED ON AN �AS IS� BASIS, WITHOUT WARRANTY OF ANY KIND, 
 10 #  EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT 
 11 #  THE COVERED SCRIPT IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR 
 12 #  PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE 
 13 #  OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE 
 14 #  DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER 
 15 #  CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION.
 16 #  NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 
 17 #  DISCLAIMER.
 18 #
 19 #  When distributing this Code, include this HEADER in each file.
 20 #  If applicable, add the following below this this HEADER, with the fields
 21 #  enclosed by brackets "[]" replaced with your own identifying information:
 22 #       Portions Copyright [yyyy] [name of copyright owner]
 23 # 
 24 #  HEADER END
 25 # 
 26 # 
 27 #       Copyright 2011 Andrew Hopkinson, Oracle Corporation UK Ltd.
 28 #
 29 ################################################################################
 30 
 31 
 32 ################################################################################
 33 #
 34 # createVLANs.sh
 35 # ==============
 36 #
 37 # Version : 1.0.0
 38 #
 39 # This scripts simply creates a number of VLANs for a specified bond. The script
 40 # must be provided with all the parameters and will then simply loop starting at
 41 # the first IP Address and incrementing by 1 until it has processed the specified 
 42 # number. 
 43 #
 44 # Parameters
 45 #
 46 #   -ip : Starting IP Address in the format 192.168.12.1.
 47 #   -nm : Netmask (i.e. 255.255.255.0)
 48 #   -c  : Number of VLANs to create
 49 #   -b  : Bond on which the VLANs should be created (bond0 or bond1)
 50 #
 51 # Version History
 52 # 1.0.0 - Initial Realease
 53 #
 54 ################################################################################
 55 
 56 
 57 # functions
 58 usage()
 59 {
 60         echo ""
 61         echo >&2 "usage: $0 [-ip Starting IP] [-nm Netmask] [-c Number of IPs require] [-b Bond]"
 62         echo""
 63 }
 64 
 65 SBIN=/sbin
 66 
 67 while [ $# -gt 0 ]
 68 do
 69         case "$1" in
 70                 -ip) STARTIP="$2"; shift;;
 71                 -nm) MASK="$2"; shift;;
 72                 -c) COUNT="$2"; shift;;
 73                 -b) BOND="$2"; shift;;
 74                 -sbin) SBIN="$2"; shift;;
 75                 *) usage; exit 1;;
 76                 *) break;;
 77         esac
 78         shift
 79 done
 80 
 81 if test "$STARTIP" = ""
 82 then
 83         echo "Starting IP Address must be specified"
 84         usage
 85         exit 1;
 86 fi
 87 if test "$MASK" = ""
 88 then
 89         echo "Netmask must be specified"
 90         usage
 91         exit 1;
 92 fi
 93 if test "$COUNT" = ""
 94 then
 95         echo "Number of IP Addresses must be specified"
 96         usage
 97         exit 1;
 98 fi
 99 if test "$BOND" = ""
100 then
101         echo "Bond must be specified"
102         usage
103         exit 1;
104 fi
105 
106 # Strip out the constituent parts of the IP Address
107 ip=$STARTIP
108 for i in 1 2 3
109 do
110         pos=`expr index $ip .`
111         if [ $i -eq 1 ]
112         then
113                 ip1=${ip:0:$pos}
114         fi
115         if [ $i -eq 2 ]
116         then
117                 ip2=${ip:0:$pos}
118         fi
119         if [ $i -eq 3 ]
120         then
121                 ip3=${ip:0:$pos}
122         fi
123         if [ $i -eq 4 ]
124         then
125                 ip4=${ip:0:$pos}
126         fi
127         ip=${ip:$pos}
128         ip4=$ip
129 done
130 
131 # Loop through and create VLANS
132 for (( i=1; i<=$COUNT; i++ ))
133 do
134         ip=$ip1$ip2$ip3$ip4
135         $SBIN/ifconfig $BOND:$i $ip netmask $MASK up
136         $SBIN/arping -q -U -c 3 -I $BOND $ip
137         ip4=`expr $ip4 + 1`
138 done
139 
140 ip4=`expr $ip4 - 1`
141 ENDIP=$ip1$ip2$ip3$ip4
142 
143 echo "Creating a ifcfg file to make permenant"
144 echo ""
145 
146 BAK_EXT=`date +"%Y%m%d-%H%M%S"`
147 IFCFG_FILE="/etc/sysconfig/network-scripts/ifcfg-$BOND"
148 VIP_RANGE_FILE="$IFCFG_FILE-range0"
149 BACKUP_VIP_RANGE_FILE=Orig.$VIP_RANGE_FILE.$BAK_EXT
150 mv $VIP_RANGE_FILE $BACKUP_VIP_RANGE_FILE
151 
152 while read line
153 do
154         f6=${line:0:6}
155         if test "$f6" = "IPADDR"
156         then
157                 echo $line
158         else
159                 echo $line >> $VIP_RANGE_FILE
160         fi
161 done < $IFCFG_FILE
162 
163 echo "IPADDR_START=$STARTIP" >> $VIP_RANGE_FILE
164 echo "IPADDR_END=$ENDIP" >> $VIP_RANGE_FILE
165 echo "CLONENUM_START=1" >> $VIP_RANGE_FILE
166 
167 echo ""
168 echo "Created VLANs $STARTIP to $ENDIP on $BOND"
169 echo ""
170 
171 

vLANs

Virtual LANs are used to combine multiple domains and hence IP subnets onto a single physical port. The be fully configured both ends of the connected cable must be capable and configured for vLANs and hence I will assume this is the case. We use vLANs to enforce security isolation, division of workload and splitting traing across multiple domains / subnets.

A simple scenario (I will document more in later articles) would be to partition the Exalogic so that we can run Production, Test and Development on the same box but keep their 10Gb Ethernet connection isolated.

To create a vLAN (no script for this one) we simply issue the createvlan command on the switch as follows.

createvlan  -vlan  -pkey default

Where:
  • ETH Port : Ethernet Port on the switch e.g. 0A-ETH-3, 1A-ETH-1, etc
  • Unique number between 2 and 4094
As long as the VLAN IDs are unique we can create multiple vLANs on each of the ports.
At this point to enforce the network issolation we will associated the newly created vLANs to specific vNICs and hence Compute Nodes. At present the scripts for creating vNICs do not do this but it can be easily achived by modifying the generated Create---VNICs.sh to add the -vlan argument to specify the required vLAN ID.


No comments:

Post a Comment

Post a Comment