[hermon] Allocate space for GRH on UD queue pairs

The Infiniband specification (volume 1, section 11.4.1.2 "Post Receive
Request") notes that for UD QPs, the GRH will be placed in the first
40 bytes of the receive buffer if present.  (If no GRH is present,
which is normal, then the first 40 bytes of the receive buffer will be
unused.)

Mellanox hardware performs this placement automatically: other headers
will be stripped (and their values returned via the CQE), but the
first 40 bytes of the data buffer will be consumed by the (probably
non-existent) GRH.

This does not fit neatly into iPXE's internal abstraction, which
expects the data buffer to represent just the data payload with the
addresses from the GRH (if present) passed as additional parameters to
ib_complete_recv().

The end result of this discrepancy is that attempts to receive
full-sized 2048-byte IPoIB packets on Mellanox hardware will fail.

Fix by allocating a separate ring buffer to hold the received GRHs.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

1 files changed, 5 insertions, 1 deletions

diff --git a/src/drivers/infiniband/hermon.h b/src/drivers/infiniband/hermon.h
index e0b028f26..61e285781 100644
--- a/src/drivers/infiniband/hermon.h
+++ b/src/drivers/infiniband/hermon.h
@@ -515,7 +515,7 @@ struct hermonprm_eth_send_wqe {
 	struct hermonprm_wqe_segment_data_ptr data[HERMON_MAX_GATHER];
 } __attribute__ (( packed ));
 
-#define HERMON_MAX_SCATTER 1
+#define HERMON_MAX_SCATTER 2
 
 struct hermonprm_recv_wqe {
 	struct hermonprm_wqe_segment_data_ptr data[HERMON_MAX_SCATTER];
@@ -686,6 +686,10 @@ struct hermon_recv_work_queue {
 	union hermon_recv_wqe *wqe;
 	/** Size of work queue */
 	size_t wqe_size;
+	/** GRH buffers (if applicable) */
+	struct ib_global_route_header *grh;
+	/** Size of GRH buffers */
+	size_t grh_size;
 	/** Doorbell record */
 	struct hermonprm_qp_db_record *doorbell;
 };