All About insert-ethers

insert-ethers is the command that allows us to manage the database of compute nodes, add compute nodes, and remove compute nodes from ROCKS. Here are a couple of interesting scenarios where you might need variations on the theme:

The best way to remove a troublesome node from the database and reinsert it after you've disciplined it:

 [root@athena0 ~]# insert-ethers --remove compute-0-8

Once you've removed the node, you can work on debugging the hardware and what not. When you want to bring it back into the cluster, use this command and wait until it's been successfully ROCK'ed:

 [root@athena0 ~]# insert-ethers --hostname compute-0-8

Another useful iteration is one that allows you to specify where to start in insert-ethers. Here we're starting with the four rack and the 27 node (U 27 as well).

 [root@athena0 ~]# insert-ethers --cabinet=4 --rank=27

Say you need to skip IP address when, for instance, the NAS device has already taken an IP and it hasn't been added to the ROCKS IP database. Before you set "baseip" check /etc/hosts for the lowest IP address in the space and increment it by one!

 [root@athena0 ~]# insert-ethers --baseip 192.168.111.88 --appliance compute --rack 4 --rank 33

If you really must manually enter a MAC address as a specific host, the proper procedure is:

 # /opt/rocks/bin/rocks add host compute-0-0 cpus=1 rack=0 rank=0 membership="Compute"
 # /opt/rocks/bin/rocks add host interface compute-0-0 eth0
 # /opt/rocks/bin/rocks set host interface ip compute-0-0 eth0 10.255.255.254
 # /opt/rocks/bin/rocks set host interface name compute-0-0 eth0 compute-0-0
 # /opt/rocks/bin/rocks set host interface mac ccompute-0-0 eth0 00:0a:5e:1a:6d:64
 # /opt/rocks/bin/rocks set host interface module compute-0-0 eth0 sk98lin
 # /opt/rocks/bin/rocks set host interface subnet compute-0-0 eth0 private
 
 
 one tricky part is the 'set host interface module' line -- you need to
 know the name of the linux driver. it may be the case that you don't
 have to specify this line and the installer will do the right thing,
 but we've never tested this case before so i can't say for sure if it
 is safe to not specify the module.
 
 after you prepopulate your database, then execute:
 
   # insert-ethers --update
 
 that will rewrite all relevant configuration files and restart all
 relevant services.
 
 (per Greg Bruno in Rocks-Discuss)

Got that? Good.. since:

 we are in the process of phasing out insert-ethers and, admittedly,
 we've been sloppy about ensuring that all the old insert-ethers flags
 are removed.
 (per Greg Bruno)

Rebuilding onto a totally uninitialized compute node disk or new MAC address:

  Power down target system ("hostname")
  On athena0:
  #insert-ethers --replace hostname
  ..will bring up the insert-ethers screen, choose "compute"
  ...insert-ethers screen will change to white-box "waiting" mode
  Power up the target, tap F12 on its console to force PXE boot
  It should receive a full-rebuild boot, the insert-ethers screen will show the progress

Extra "nodes" showing up even after doing a 'remove':

 Edit the file  /opt/torque/server_priv/nodes  to remove the entry