vii
8.1.27 Replacing a Voltaire InfiniBand switch.................................................................................. 8-15
8.1.28 Relocating an InfiniBand cable to a different port on the InfiniBand switch ................................ 8-15
8.1.29 Replacing a Power Distribution Unit (PDU) on a rack .............................................................. 8-16
8.1.30 Replacing a Power Distribution Module (AC power strip) on a rack.......................................... 8-16
8.1.31 Replacing a Power Distribution Unit (PDU) on an HP ProCurve Switch 2650 or HP ProCurve
Switch 2626 ..................................................................................................................... 8-16
8.2 Upgrading firmware ............................................................................................................... 8-16
8.2.1
Upgrading firmware on a server.......................................................................................... 8-16
8.2.1.1
Upgrading online using the OnlineROM Flash Component executable................................. 8-17
8.2.1.2
Upgrading offline using a USB pen drive ......................................................................... 8-17
8.2.1.3
Upgrading offline using a floppy disk drive — G3 servers only........................................... 8-17
8.2.2
Upgrading firmware on Smart Array 6404 adapters and SFS20 arrays ................................... 8-18
8.2.3
Upgrading firmware on an InfiniBand adapter ...................................................................... 8-18
8.2.3.1
Upgrading the firmware on a Voltaire PCI-X HCA adapter ................................................. 8-19
8.3 Adding and removing components ........................................................................................... 8-20
8.3.1
Adding Object Storage Servers ........................................................................................... 8-20
8.3.2
Removing Object Storage Servers........................................................................................ 8-22
8.3.3
Adding SFS20 arrays......................................................................................................... 8-24
8.3.4
Removing SFS20 arrays...................................................................................................... 8-24
8.3.5
Adding a dual or a bonded Gigabit Ethernet interconnect ...................................................... 8-24
8.3.6
Removing a dual or a bonded Gigabit Ethernet interconnect ................................................... 8-26
9 Troubleshooting
9.1 Server fails to boot during installation ......................................................................................... 9-3
9.2 Server fails to boot.................................................................................................................... 9-3
9.3 Server stops responding ............................................................................................................ 9-3
9.4 Server fails to mount administration LUN — EXT3-fs error .............................................................. 9-4
9.5 Benign message displayed by the configure system command........................................................ 9-4
9.6 The configure server command fails because static routes are incorrectly configured......................... 9-4
9.7 Server with Quadrics interconnect may fail to boot ....................................................................... 9-5
9.8 The scan mac command fails ..................................................................................................... 9-5
9.9 Setting Lustre debug level .......................................................................................................... 9-6
9.10 System database has become corrupted...................................................................................... 9-6
9.11 Service LUN has become corrupted ............................................................................................ 9-6
9.12 Replacing a service LUN with a spare service LUN ....................................................................... 9-7
9.13 Booting a server in single-user mode ........................................................................................... 9-8
9.14 The SFS CLI reports that the database is not ready........................................................................ 9-8
9.15 The configure array command fails............................................................................................. 9-9
9.15.1 SFS20 array cabling problems .............................................................................................. 9-9
9.15.2 Preferred server for the SFS20 array is down ........................................................................ 9-10
9.16 Command remains in the Allocated state................................................................................... 9-10
9.17 The configure server command remains in the Unallocated state................................................... 9-10
9.18 Emergency clustat events occur during configure server command ................................................ 9-11
9.19 The show log command fails with corrupted record .................................................................... 9-11
9.20 SFS CLI command fails with error: Could not insert duplicate record ............................................. 9-11
9.21 Press "F1" key to continue prompt ............................................................................................ 9-12
9.22 Gigabit Ethernet -113 error code.............................................................................................. 9-12
9.23 Troubleshooting the Quadrics interconnect................................................................................. 9-13
9.23.1 Start, stop, and loading problems ........................................................................................ 9-13
9.23.2 Nodeset and Node ID information ....................................................................................... 9-14
9.23.3 Checking active Lustre communications over the Quadrics interconnect .................................... 9-14
9.23.4 Gathering debugging information........................................................................................ 9-14
9.24 Troubleshooting the Myrinet interconnect................................................................................... 9-15
9.24.1 Start, stop, and check status ................................................................................................ 9-15
9.24.2 Myrinet interconnect stress test............................................................................................. 9-15
9.24.3 Nodeset and Node ID information ....................................................................................... 9-15
9.24.4 Disconnected link in Myrinet 2XP interconnect ....................................................................... 9-15
9.25 Troubleshooting the Voltaire InfiniBand interconnect.................................................................... 9-16
9.25.1 Start, stop, and loading problems ........................................................................................ 9-16
