[tls] Split received records over multiple I/O buffers
TLS servers are not obliged to implement the RFC3546 maximum fragment
length extension, and many common servers (including OpenSSL, as used
in Apache's mod_ssl) do not do so. iPXE may therefore have to cope
with TLS records of up to 16kB. Allocations for 16kB have a
non-negligible chance of failing, causing the TLS connection to abort.
Fix by maintaining the received record as a linked list of I/O
buffers, rather than a single contiguous buffer. To reduce memory
pressure, we also decrypt in situ, and deliver the decrypted data via
xfer_deliver_iob() rather than xfer_deliver_raw().
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[ipoib] Expose Ethernet-compatible eIPoIB link-layer addresses and headers
Almost all clients of the raw-packet interfaces (UNDI and SNP) can
handle only Ethernet link layers. Expose an Ethernet-compatible link
layer to local clients, while remaining compatible with IPoIB on the
wire. This requires manipulation of ARP (but not DHCP) packets within
the IPoIB driver.
This is ugly, but it's the only viable way to allow IPoIB devices to
be driven via the raw-packet interfaces.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Commit 501527d ("[http] Treat any unexpected connection close as an
error") introduced a regression causing HTTP SAN booting to fail. At
the end of the response to the HEAD request, the call to http_done()
would erroneously believe that the server had disconnected in the
middle of the HTTP headers.
Fix by treating the header block from a HEAD request as a trailer
block. This fixes the problem and also simplifies the logic in
http_rx_header().
Reported-by: Shao Miller <shao.miller@yrdsb.edu.on.ca>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The FTP SIZE command allows us to get the size of a particular file,
as a consequence, we can now show proper transfer progression while
fetching a file using the FTP protocol.
Signed-off-by: Marin Hannache <git@mareo.fr>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Treat any unexpected connection close as an error
iPXE currently checks that the server has not closed the connection
mid-stream (i.e. in the middle of a chunked transfer, or before the
specified Content-Length has been received), but does not check that
the server got as far as starting to send data. Consequently, if the
server closes the connection before any data is transferred (e.g. if
the server gives up waiting while iPXE performs the validation steps
for TLS), then iPXE will treat this as a successful transfer of a
zero-length file.
Fix by checking the RX connection state, and forcing an error if the
server has closed the connection at an unexpected point.
Originally-fixed-by: Marin Hannache <mareo@mareo.fr>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[tcp] Truncate TCP window to prevent future packet discards
Whenever memory pressure causes a queued packet to be discarded (and
so retransmitted), reduce the maximum TCP window to a size that would
have prevented the discard.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Discarding the active ARP cache entry in the middle of a download will
substantially disrupt the TCP stream. Try to minimise any such
disruption by treating ARP cache entries as expensive, and discarding
them only when nothing else is available to discard.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[netdevice] Process all received packets in net_poll()
The current logic is to process at most one received packet per call
to net_poll(), on the basis that refilling the hardware descriptor
ring should be delayed as little as possible. However, this limits
the rate at which packets can be processed and ultimately ends up
adding latency which, in turn, limits the achievable throughput.
With temporary modifications in place to essentially remove all
resource constraints (heap size increased to 16MB, RX descriptor ring
increased to 64 descriptors) and a TCP window size of 1MB, the
throughput on a gigabit (i.e. 119MBps) network can be observed to fall
off exponentially from around 115MBps to around 75MBps. Changing
net_poll() to process all received packets results in a steady
119MBps throughput.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[arp] Prevent ARP cache entries from being deleted mid-transmission
Each ARP cache entry maintains a transmission queue, which is sent out
as soon as the link-layer address is known. If multiple packets are
queued, then it is possible for memory pressure to cause the ARP cache
discarder to be invoked during transmission of the first packet, which
may cause the ARP cache entry to be deleted before the second packet
can be sent. This results in an invalid pointer dereference.
Avoid this problem by reference-counting ARP cache entries and
ensuring that an extra reference is held while processing the
transmission queue, and by using list_first_entry() rather than
list_for_each_entry_safe() to traverse the queue.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Commit ea61075 ("[tcp] Add support for TCP window scaling") introduced
a potential NULL pointer dereference by referring to the connection's
send window scale before checking whether or not the connection is
known.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[tls] Request a maximum fragment length of 2048 bytes
The default maximum plaintext fragment length for TLS is 16kB, which
is a substantial amount of memory for iPXE to have to allocate for a
temporary decryption buffer.
Reduce the memory footprint of TLS connections by requesting a maximum
fragment length of 2kB.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The maximum unscaled TCP window (64kB) implies a maximum bandwidth of
around 300kB/s on a WAN link with an RTT of 200ms. Add support for
the TCP window scaling option to remove this upper limit.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[tcpip] Allow for architecture-specific TCP/IP checksum routines
Calculating the TCP/IP checksum on received packets accounts for a
substantial fraction of the response latency.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[dhcp] Request broadcast responses when we already have an IPv4 address
FCoE requires the use of multiple local unicast link-layer addresses.
To avoid the complexity of managing multiple addresses, iPXE operates
in promiscuous mode. As a consequence, any unicast packets with
non-matching IPv4 addresses are rejected at the IPv4 layer (rather
than at the link layer).
This can cause problems when issuing a second DHCP request: if the
address chosen by the DHCP server does not match the existing address,
then the DHCP response will itself be rejected.
Fix by requesting a broadcast response from the DHCP server if the
network interface already has any IPv4 addresses.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Provide credentials only when requested by server
Provide HTTP Basic authentication credentials only in response to a
401 Unauthorized response from the server.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Defer processing response code until after receiving all headers
Some headers can modify the meaning of the response code. For
example, a WWW-Authenticate header can change the interpretation of a
401 Unauthorized response from "Access denied" to "Please
authenticate".
Signed-off-by: Michael Brown <mcb30@ipxe.org>
iSCSI generally includes a full SCSI response only when an error
occurs. iscsi_scsi_done() currently passes the NULL response through
to scsi_response(), which ends up causing scsicmd_response() to
dereference a NULL pointer.
Fix by calling scsi_response() only if we have a non-NULL response.
Reported-by: Brendon Walsh <brendonwalsh@niamu.com>
Tested-by: Brendon Walsh <brendonwalsh@niamu.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
X.509 certificate processing currently produces an overwhelming amount
of debugging information. Move some of this from DBGLVL_LOG to
DBGLVL_EXTRA, to make the output more manageable.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Automatically attempt to download any required cross-signing
certificates from http://ca.ipxe.org/auto, in order to enable the use
of standard SSL certificates issued by public CAs.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
To allow for automatic download of cross-signing certificates and for
OCSP, the validation of certificates must be an asynchronous process.
Create a stub validator which uses a job-control interface to report
the result of certificate validation.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[crypto] Allow certificate chains to be long-lived data structures
At present, certificate chain validation is treated as an
instantaneous process that can be carried out using only data that is
already in memory. This model does not allow for validation to
include non-instantaneous steps, such as downloading a cross-signing
certificate, or determining certificate revocation status via OCSP.
Redesign the internal representation of certificate chains to allow
chains to outlive the scope of the original source of certificates
(such as a TLS Certificate record).
Allow for certificates to be cached, so that each certificate needs to
be validated only once.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Avoid using stack-allocated memory in http_step()
http_step() allocates a potentially large block of storage (since the
URI can be arbitrarily long), and can be invoked as part of an already
deep call stack via xfer_window_changed().
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Michael Brown