[infiniband] Return status code from ib_create_cq() and ib_create_qp()
Any underlying errors arising during ib_create_cq() or ib_create_qp()
are lost since the functions simply return NULL on error. This makes
debugging harder, since a debug-enabled build is required to discover
the root cause of the error.
Fix by returning a status code from these functions, thereby allowing
any underlying errors to be propagated.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Avoid potential integer overflow when calculating memory mappings
When the area to be mapped straddles the 2GB boundary, the expression
(high+size) will overflow on the first loop iteration. Fix by using
(end-size), which cannot underflow.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The Infiniband specification (volume 1, section 11.4.1.2 "Post Receive
Request") notes that for UD QPs, the GRH will be placed in the first
40 bytes of the receive buffer if present. (If no GRH is present,
which is normal, then the first 40 bytes of the receive buffer will be
unused.)
Mellanox hardware performs this placement automatically: other headers
will be stripped (and their values returned via the CQE), but the
first 40 bytes of the data buffer will be consumed by the (probably
non-existent) GRH.
This does not fit neatly into iPXE's internal abstraction, which
expects the data buffer to represent just the data payload with the
addresses from the GRH (if present) passed as additional parameters to
ib_complete_recv().
The end result of this discrepancy is that attempts to receive
full-sized 2048-byte IPoIB packets on Mellanox hardware will fail.
Fix by allocating a separate ring buffer to hold the received GRHs.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Ensure hardware is quiescent when no interfaces are open
WinPE has been observed to call PXENV_UNDI_SHUTDOWN but not
PXENV_STOP_UNDI. This means that Hermon hardware is left partially
active (firmware running and one event queue mapped) when WinPE starts
up, which can cause a Blue Screen of Death.
Fix by ensuring that the hardware is left quiescent (with the firmware
stopped) when no interfaces are open.
Reported-by: Itay Gazit <itayg@mellanox.co.il>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Remove temporary workaround for missing BOFM mport support
This reverts commit 15c1200 ("[hermon] Work around missing mport
support in current BOFM implementations").
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Work around missing mport support in current BOFM implementations
Current BOFM versions are unable to create entries with mport>1, which
means that only the port 1 MAC address can be explicitly specified.
Work around this by using the provided MAC address as a base address
for all subsequent ports. For example, if BOFM assigns the address
00:1A:64:76:00:09 for port 1
then we will assign the addresses
00:1A:64:76:00:09 for port 1
00:1A:64:76:00:0a for port 2
Future BOFM versions that may correctly support mport will work with
this scheme without modification provided that the BOFM entries are
created in increasing order of mport. Since BOFM tools tend to
generate entries in increasing order (of slot, port, etc), this is not
an unreasonable compromise.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Avoid memory leak of untreated events by having circular event queue
operation.
Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add PRM structures to support Hermon Ethernet devices.
Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Unlike Arbel, port parameters must be applied via a separate call to
SET_PORT, rather than as parameters to INIT_PORT.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Minimise the number of VPM mapping operations
Mapping a single page at a time causes a several-second delay at
device initialisation time. Reduce this by mapping multiple pages at
a time, using the largest block sizes possible given the alignment
constraints.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Only port state change events are currently mapped to our event queue,
since those are the only events we are prepared to handle. This
ignores a potentially useful source of diagnostic information in the
case of unexpected failures.
Fix by mapping all events to the event queue; a build with debugging
enabled will therefore at least dump the raw content of the unexpected
events.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
No event is generated upon reaching INIT, so we must poll separately
for link state changes while we remain DOWN.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[infiniband] Always call ib_link_state_changed() in ib_smc_update()
ib_smc_update() potentially updates the Infiniband port state, and so
should almost always be followed by a call to ib_link_state_changed().
The one exception is the call made to ib_smc_update() before the
device is registered.
Fix by removing explicit calls to ib_link_state_changed() from drivers
using ib_smc_update(), including a call to ib_link_state_changed()
within ib_smc_update(), and creating a separate ib_smc_init() for use
prior to device registration.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Use correct alignment for doorbell records
Doorbell records are currently embedded within the completion queue
and receive work queue strucures, which are allocated using zalloc()
and so have an alignment guarantee of only sizeof(void*), i.e. four
bytes. This is sufficient for the receive work queue, but not for the
completion queue, which requires an alignment guarantee of eight
bytes.
Though not guaranteed, it so happens that zalloc() will always return
a pointer that is exactly four bytes above a sixteen-byte boundary.
The completion queue doorbell record is therefore always misaligned,
and the value passed to the hardware via SW2HW_CQ is actually always
pointing to the page_offset value within the MTT descriptor (which
directly precedes the inline doorbell record). Provided that the page
offset is greater than 0x100, this looks to the hardware like an
update_ci value of greater than 0x010000 (taking into account
endianness differences), and so the hardware will happily deliver more
than 0x010000 completions before stopping. Hence this problem is
rarely observable.
Fix by allocating the doorbell records separately and using the
correct alignment constraints.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Set event queue number for completion queues
Give completion queues a chance to deliver exception events by
programming in the number of our event queue (currently used only for
port state changes).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Improve the utility of debugging messages by including the relevant
port number, queue number (QPN, CQN, EQN), work queue entry (WQE)
number, and physical addresses wherever applicable.
Add hermon_dump_cqctx() for dumping a completion queue context, and
hermon_fill_nop_send_wqe() for inserting NOPs into send work queues.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Access to the gpxe.org and etherboot.org domains and associated
resources has been revoked by the registrant of the domain. Work
around this problem by renaming project from gPXE to iPXE, and
updating URLs to match.
Also update README, LOG and COPYRIGHTS to remove obsolete information.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[hermon] Change hermon_alloc_icm() to conform to ConnectX2 requirements
Align each ICM member alloc to the member size instead of page size.
Increase multicast table size to 128.
Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Signed-off-by: Marty Connor <mdc@etherboot.org>
Some systems will retry their boot sequence in the event of a boot
failure. On these systems, the second and subsequent boot attempts
will fail to initialise the Hermon HCA.
Fix by resetting the HCA during probe(). This incurs a one-second
cost, but there seems to be no viable alternative.
Originally-fixed-by: Itay Gazit <itaygazit@gmail.com>
The ACK timeout determines how long we take to notice a failed
Reliable Connection. Reducing it from the arbitrary value of 19 down
to 14 reduces the individual ACK timeout from around 2.1s to 67ms;
this in turn reduces the time to tear down and re-establish a broken
SRP session from around 30s to around 1s.
[hermon] Randomise the high-order bits of queue pair numbers
The Infiniband Communication Manager will refuse to establish a
connection if it believes the connection is already established.
There is no immediately obvious way to ask it to tear down the
existing connection and replace it; to issue a DREP we would need to
know the local and remote communication IDs used for the previous
connection setup.
We can work around this by randomising the high-order bits of the
queue pair number; these have no significance to the hardware, but are
sufficient to convince the IB CM that this is a different connection.
[infiniband] Change IB_{QPN,QKEY,QPT} names from {SMA,GMA} to {SMI,GSI}
The IBA specification refers to management "interfaces" and "agents".
The interface is the component that connects to the queue pair and
sends and receives MADs; the agent is the component that constructs
the reply to the MAD.
Rename the IB_{QPN,QKEY,QPT} constants as a first step towards making
this separation in gPXE.
[hermon] Allow software GMA to receive packets destined for QP1
The Linux IB Communication Manager will always send MADs to QP1,
rather than back to the originating QP. On Hermon, QP1 is by default
handled by the embedded firmware. We can change this, but the cost is
that we have to handle both QP0 and QP1 (i.e. we have to provide SMA
as well as GMA service in software), and we have to use MLX queues
rather than standard UD queues (i.e. we have to construct the UD
datagrams by hand).
There doesn't seem to be any viable way around this situation, ugly
though it is.
Michael Brown