[http] Rewrite HTTP core to support content encodings
Rewrite the HTTP core to allow for the addition of arbitrary content
encoding mechanisms, such as PeerDist and gzip.
The core now exposes http_open() which can be used to create requests
with an explicitly selected HTTP method, an optional requested content
range, and an optional request body. A simple wrapper provides the
preexisting behaviour of creating either a GET request or an
application/x-www-form-urlencoded POST request (if the URI includes
parameters).
The HTTP SAN interface is now implemented using the generic block
device translator. Individual blocks are requested using http_open()
to create a range request.
Server connections are now managed via a connection pool; this allows
for multiple requests to the same server (e.g. for SAN blocks) to be
completely unaware of each other. Repeated HTTPS connections to the
same server can reuse a pooled connection, avoiding the per-connection
overhead of establishing a TLS session (which can take several seconds
if using a client certificate).
Support for HTTP SAN booting and for the Basic and Digest
authentication schemes is now optional and can be controlled via the
SANBOOT_PROTO_HTTP, HTTP_AUTH_BASIC, and HTTP_AUTH_DIGEST build
configuration options in config/general.h.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
iscsi_tx_done() is missing "break" statements at the end of each case.
(Fortunately, this happens not to cause a bug in practice, since
iscsi_login_request_done() is effectively a no-op when completing a
data-out PDU.)
Reported-by: Wissam Shoukair <wissams@mellanox.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[base16] Add buffer size parameter to base16_encode() and base16_decode()
The current API for Base16 (and Base64) encoding requires the caller
to always provide sufficient buffer space. This prevents the use of
the generic encoding/decoding functionality in some situations, such
as in formatting the hex setting types.
Implement a generic hex_encode() (based on the existing
format_hex_setting()), implement base16_encode() and base16_decode()
in terms of the more generic hex_encode() and hex_decode(), and update
all callers to provide the additional buffer length parameter.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Microsoft IIS supports only MD5-sess for Digest authentication.
Requested-by: Andreas Hammarskjöld <junior@2PintSoftware.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[legal] Relicense files under GPL2_OR_LATER_OR_UBDL
These files cannot be automatically relicensed by util/relicense.pl
since they either contain unusual but trivial contributions (such as
the addition of __nonnull function attributes), or contain lines
dating back to the initial git revision (and so require manual
knowledge of the code's origin).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[build] Expose build timestamp, build name, and product names
Expose the build timestamp (measured in seconds since the Epoch) and
the build name (e.g. "rtl8139.rom" or "ipxe.efi"), and provide the
product name and product short name in a single centralised location.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Parse the sense data to extract the reponse code, the sense key, the
additional sense code, and the additional sense code qualifier.
Originally-implemented-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Accept Content-Length header with trailing whitespace
At least one HTTP server (Google's OCSP responder) has been observed
to generate a Content-Length header with trailing whitespace.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Automatically retry request on a 503 Service Unavailable
A web server may return a 503 Service Unavailable response along with
a Retry-After header to direct the client to retry the request at a
later time.
The Retry-After header may be a number of seconds, or a full HTTP
timestamp (e.g. "Fri, 7 Mar 2014 17:22:14 GMT"). We have no
reasonable way of parsing a full HTTP timestamp; if the server chooses
to use this format then we simply retry after a fixed 5-second delay.
As per RFC 2616, in the absence of a Retry-After header we treat a
status code of 503 Service Unavailable as being equivalent to 500
Internal Server Error, and immediately fail the request.
Requested-by: Suresh Sundriyal <ssundriy@vmware.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add support for parsing of URIs containing literal IPv6 addresses
(e.g. "http://[fe80::69ff:fe50:5845%25net0]/boot.ipxe").
Duplicate URIs by directly copying the relevant fields, rather than by
formatting and reparsing a URI string. This relaxes the requirements
on the URI formatting code and allows it to focus on generating
human-readable URIs (e.g. by not escaping ':' characters within
literal IPv6 addresses). As a side-effect, this allows relative URIs
containing parameter lists (e.g. "../boot.php##params") to function
as expected.
Add validity check for FTP paths to ensure that only printable
characters are accepted (since FTP is a human-readable line-based
protocol with no support for character escaping).
Construct TFTP next-server+filename URIs directly, rather than parsing
a constructed "tftp://..." string,
Add self-tests for URI functions.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Note that IANA has not yet assigned a DHCPv6 option code for the
syslog server. When a code is assigned, the definition of
DHCPV6_LOG_SERVERS should be updated. Until then, an IPv6 address of
a syslog server can be configured manually using e.g.
set syslog6 3ffe:302:11:2::8309
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[settings] Explicitly separate the concept of a completed fetched setting
The fetch_setting() family of functions may currently modify the
definition of the specified setting (e.g. to add missing type
information). Clean up this interface by requiring callers to provide
an explicit buffer to contain the completed definition of the fetched
setting, if required.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Allow HTTP POST requests to be generated when the URI includes a
parameter list. For example:
#!ipxe
params
param mac ${net0/mac}
param uuid ${uuid}
param asset ${asset}
chain http://boot.ipxe.org/demo/boot.php##params
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Commit 501527d ("[http] Treat any unexpected connection close as an
error") introduced a regression causing HTTP SAN booting to fail. At
the end of the response to the HEAD request, the call to http_done()
would erroneously believe that the server had disconnected in the
middle of the HTTP headers.
Fix by treating the header block from a HEAD request as a trailer
block. This fixes the problem and also simplifies the logic in
http_rx_header().
Reported-by: Shao Miller <shao.miller@yrdsb.edu.on.ca>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The FTP SIZE command allows us to get the size of a particular file,
as a consequence, we can now show proper transfer progression while
fetching a file using the FTP protocol.
Signed-off-by: Marin Hannache <git@mareo.fr>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Treat any unexpected connection close as an error
iPXE currently checks that the server has not closed the connection
mid-stream (i.e. in the middle of a chunked transfer, or before the
specified Content-Length has been received), but does not check that
the server got as far as starting to send data. Consequently, if the
server closes the connection before any data is transferred (e.g. if
the server gives up waiting while iPXE performs the validation steps
for TLS), then iPXE will treat this as a successful transfer of a
zero-length file.
Fix by checking the RX connection state, and forcing an error if the
server has closed the connection at an unexpected point.
Originally-fixed-by: Marin Hannache <mareo@mareo.fr>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Provide credentials only when requested by server
Provide HTTP Basic authentication credentials only in response to a
401 Unauthorized response from the server.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Defer processing response code until after receiving all headers
Some headers can modify the meaning of the response code. For
example, a WWW-Authenticate header can change the interpretation of a
401 Unauthorized response from "Access denied" to "Please
authenticate".
Signed-off-by: Michael Brown <mcb30@ipxe.org>
iSCSI generally includes a full SCSI response only when an error
occurs. iscsi_scsi_done() currently passes the NULL response through
to scsi_response(), which ends up causing scsicmd_response() to
dereference a NULL pointer.
Fix by calling scsi_response() only if we have a non-NULL response.
Reported-by: Brendon Walsh <brendonwalsh@niamu.com>
Tested-by: Brendon Walsh <brendonwalsh@niamu.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[http] Avoid using stack-allocated memory in http_step()
http_step() allocates a potentially large block of storage (since the
URI can be arbitrarily long), and can be invoked as part of an already
deep call stack via xfer_window_changed().
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[iscsi] Send any padding inline with the data segment
Some iSCSI targets respond to a PDU before receiving the padding
bytes. If the target responds quickly enough, this can cause iPXE to
start processing a new TX PDU before the padding bytes have been sent,
which results in a protocol violation.
Fix by always transmitting the padding bytes along with the data
segment.
Originally-fixed-by: Shyam Iyer <shyam_iyer@dell.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
As RFC 2616 10.3.4 explains, a 303 status is the proper HTTP 1.1
behavior for what most HTTP 1.0 clients did with code 302.
Signed-off-by: Jason Lunz <lunz@acm.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Separate out the core HTTP functionality (which is shared by both HTTP
and HTTPS) from the provision of the "http://" URI opener. This
allows for builds that support only "https://" URIs.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
[iscsi] Fail immediately if target rejects any of our parameters
Some iSCSI targets (observed with stgt) can be configured to reject
connections that do not use header or data digests, and will respond
with "HeaderDigest=Reject" and/or "DataDigest=Reject", while still
allowing the connection to proceed to the full feature phase.
According to a strict reading of RFC3720, we are perfectly safe to
ignore these "Reject" messages: upon such a rejection "the negotiated
key is left at its current value (or default if no value was set)".
Since the default value for both HeaderDigest and DataDigest is
"None", then the only viable conclusion to be drawn is that the value
resulting from "Reject" is still "None".
Unfortunately, stgt doesn't seem to agree with this interpretation of
events, causing us to eventually report an unhelpful "connection timed
out" message to the user when we don't get any response to our first
PDU in full feature phase.
Fix by detecting any rejected parameters and immediately reporting an
error, which at least gives the user some insight as to what the real
problem may be.
Reported-by: Michal Suchanek <hramrach@centrum.cz>
Tested-by: Michal Suchanek <hramrach@centrum.cz>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
According to section 14.23 of RFC2616, an HTTP Host header without
port implies the default port is used. Thus, when fetching from
anywhere but port 80 for HTTP or 443 for HTTPS, the port ought to be
explicitly given in that header. Otherwise, some servers might fail
to associate the request with the correct virtual host or generate
incorrect self-referencing URLs.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The iSCSI TX process can now be woken up by the TCP socket via
xfer_window_changed(), so it is no longer valid to assume that
iscsi_tx_step() can be called in state ISCSI_TX_IDLE only immediately
after completing a transmission.
Fix by calling iscsi_tx_done() only upon a transition into state
ISCSI_TX_IDLE.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Provide support for HTTP range requests, and expose this functionality
via the iPXE block device API. This allows SAN booting from a root
path such as:
sanboot http://boot.ipxe.org/freedos/fdfullcd.iso
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Michael Brown