.Boot over IB over Mellanox HCAs README - Pre-Alpha release ========================================================== Document #2442, Rev 0.10, December 2005 1. General ----------- This README describes the Boot over IB package which enables booting a Linux kernel from a remote server using one of the Mellanox Technologies HCA devices. The package is based on Etherboot 5.4.1. The package actually implements a network driver for Etherboot. The wire protocol is compliant with IP Over IB (see http://www.ietf.org/html.charters/ipoib-charter.html for related documents). Etherboot uses a traditional setup of a DHCP server and a TFTP server to perform a remote boot in a similar manner to Ethernet NICs. The binary code is exported by the device as an expansion ROM image. 2. Supported Devices --------------------- The following Mellanox Technologies HCA devices are supported: PCI Device ID Mellanox HCA Device ---------------------------------------------- 23108 InfiniHost (P/N MT23108) 25208 InfiniHost III Ex (P/N MT25208) (InfiniHost) 25218 InfiniHost III Ex (P/N MT25208) (MemFree) 25204 InfiniHost III Lx (P/N MT25204) Note: For devices with more than one IB port, port 1 is used for communications. 3. Compiling ---------------- From the src directory: Run" make bin/. where device can be any of: MT23108 MT25218 MT25208 MT25204 and ext can be rom, zrom etc. (see Etherbot doumentation for more details) 4. Directory Structure ----------------------- All driver files are available under src/drivers/net/mlx_ipoib/. Under this directory the following files can be found: *** doc - Contains related documents including this file. *** patches - Contains needed patch files. *** samples - Contains sample files. *** . Contains driver source files. 5. Burning the Flash Memory ---------------------------- The binary code resides in the same Flash device of the device firmware. However the binary files are distinct and do not affect each other. Mellanox's 'mlxburn' tool is available for burning, however, it is not possible to burn the expansion ROM image by itself; rather, both the firmware and expansion ROM images must be burnt simultaneously. 'mlxburn' is part of the Mellanox Firmware Tools (MFT) package available for download from www.mellanox.com under 'Firmware Downloads'. Example: The following command burns a firmware image and an expansion ROM image to an InfiniHost Adapter Card (P/N MHX-CE128-T.ini): mlxburn -fw fw-23108-a1-rel.mlx -exp_rom MT23108.bin /dev/mst/mt23108_pci_cr0 -conf MHX-CE128-T.ini *** Important Note: The .ini file must support burning expansion ROMs. For example, the following lines must appear in the .ini file. If they do not, please add them manually. [ADAPTER] exp_rom_en = true Mellanox's web site contains firmware binary files with extension .bin.gz. These files contain contain EVERYTHING the goes in the flash memory and thus are NOT suitable for including the expansion rom image. Therefore, one use the files with .mlx extension also available from Mellanox's web site. 6. Preparing the DHCP Server ----------------------------- The DHCP server may need to be modified in order to work on IPOIB. Some distributuions alreay support this (Some SUSE distributuions) while others do not. If the pre-installed server does not support IPOIB, the user can download the sources from ISC http://www.isc.org/ and apply the appropriate patch in the patches directory. The DHCP server must run on a machine which supports IP Over IB. The Mellanox IBGD package (gen1 or gen2) can be used to provide this. To run the DHCP server from the command line use: dhcpd ib0 7. DHCP Server Configuration File ---------------------------------- In order for the DHCP server to provide configuration records for clients, an appropriate configuration file dhcpd.conf must be created and put under /etc/. A sample configuration file with comments can be found in the samples directory. 8. OpenSM ---------- To successfully boot over IB, the IB network must contain a Subnet Manager which configures the IB network. OpenSM is part of the IBGD distribution and can be used to accomplish that. Note that OpenSM may be run on the same host running the DHCP server but it is not mandatory. 9. TFTP Server --------------- When the DHCP session is completed, the client has the IP address of the TFTP server from which it should download the kernel image. This TFTP server must run on a machine with InfiniBand support. The client loads first a loader image based on PXE API which then loads the kernel image. The image can be found in the Linux kernel homepage: http://www.kernel.org/pub/linux/boot/syslinux/ 10. BIOS Configuration ----------------------- The expansion ROM image presents itself to the BIOS as a boot device. As a result, the BIOS will add it to the list of boot devices. The priority of this list can be modified when entering the BIOS setup. The boot over IB image must be set first for the BIOS to attempt to use it first. 11. Operation -------------- When booting the client, a message appears on the screen describing the device found and the revision of the code. The user has 3 seconds to press a key for increased functionality: 'V' will increase verbosity. 'I' will prinit some information to aid in configuring the DHCP configuration file. In this case the display will hold for 5 seconds to let the user grasp the information. Note that increasing verbosity will significantly slow boot time and will print lots of debug messages to the screen. 12. Diskless Machines ---------------------- Most often it is required to boot a diskless machine. In these cases the kernel mounts its root file system over NFS over one of the interfaces. For this to happen on a client with only InfiniBand interfaces, the kernel image must be configured accordingly and must include IP Over IB support. This can be achieved either by compiling this into the kernel or using initrd images that contain IP Over IB support. 13. Changing Defaults ---------------------- As stated the driver uses port 1 for its communications. To use another port edit the file src/drivers/net/ib_driver.h and modify the definition of PXE_IB_PORT. 14. Installing a package from Mellanox -------------------------------------- The package comes as a compressed file with extension .bz2 or .gz. Follow these steps: 1. Create a directory 2. cd to this directory 3. tar jxf for .bz2 files or tar zxf for .gz files The binaries can be found under src/bin When using a package obtained from Mellanox Technologies' web site, the directory src/bin will contain the driver binary files. The files have a .bin extension and are equivalent to the same files with .zrom extension.