The (very approximate) split between Makefile.housekeeping and
Makefile is that the former provides mechanism and the latter provides
policy.

Provide a section within Makefile as a home for predefined build
shortcuts such as the existing all-drivers build.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The WORKAROUND_CFLAGS list is constructed based on running tests on
the target compiler, and the results may not be valid for the host
compiler.

The only relevant workaround required for the host compiler is
-Wno-stringop-truncation, which is needed to avoid a spurious compiler
warning for a totally correct usage of strncpy() in util/elf2efi.c.

Duplicating the workaround tests for the host compiler is messy, as is
conditionally applying __attribute__((nonstring)).  Fix instead by
disapplying WORKAROUND_CFLAGS for the host compiler, and using
memcpy() with an explicitly calculated length instead of strncpy() in
util/elf2efi.c.

Reported-by: Ignat Korchagin <ignat@cloudflare.com>
Reported-by: Christopher Clark <christopher.w.clark@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Compiling with gcc 9.1 generates lots of "taking address of packed
member of ... may result in an unaligned pointer value" warnings.

Some of these warnings are genuine, and indicate correctly that parts
of iPXE currently require the CPU (or runtime environment) to support
unaligned accesses.  For example: the TCP/IP receive data path will
attempt to access 32-bit fields that may not be aligned to a 32-bit
boundary.

Other warnings are either spurious (such as when the pointer is to a
variable-length byte array, which can have no alignment requirement
anyway) or unhelpful (such as when the pointer is used solely to
provide a debug colour value for the DBGC() macro).

There appears to be no easy way to silence the spurious warnings.
Since the ability to perform unaligned accesses is already a
requirement for iPXE, work around the problem by silencing this class
of warnings.

Signed-off-by: Valentine Barshak <gvaxon@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Some versions of gcc seem to silently accept an attempt to disable an
unrecognised warning (e.g. via -Wno-stringop-truncation) but will then
report the unrecognised warning if any other error occurs during the
build, resulting in a potentially misleading error message.

Avoid this potential confusion by using the positive-form tests in
order to determine the workaround CFLAGS.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The gcc 8 compiler introduces a warning for certain string
manipulation functions, flagging usages which _may_ not be intended.
An audit of the iPXE sources indicates all usages of strncat and
strncpy are as intended, so the warnings currently issued are not
helpful, especially if warnings are considered errors.

Fix by detecting gcc's support for -Wno-stringop-truncation and, if
detected, using that option to avoid the warning.

Signed-off-by: Bruce Rogers <brogers@suse.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Also-fixed-by: Christian Hesse <list@eworm.de>
Also-fixed-by: Roman Kagan <rkagan@virtuozzo.com>
Also-fixed-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
Also-fixed-by: Olaf Hering <olaf@aepfle.de>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

When submitting binaries for UEFI Secure Boot signing, certain
known-dubious subsystems (such as 802.11 and NFS) must be excluded
from the build.  Mark the directories containing these subsystems as
insecure, and allow the build target to include an explicit "security
flag" (a literal "-sb" appended to the build platform) to exclude
these source directories from the build process.

For example:

  make bin-x86_64-efi-sb/ipxe.efi

will build iPXE with all code from the 802.11 and NFS subsystems
excluded from the build.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

As with assertions, profiling is enabled for objects built with any
debug level (including an explicit debug level of zero).

Allow profiling to be globally enabled or disabled by adding PROFILE=1
or PROFILE=0 respectively to the build command line.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Assertions are enabled for objects built with any debug level
(including an explicit debug level of zero).  It is sometimes useful
to be able to enable assertions across all objects; this currently
requires manually hacking include/assert.h.

Allow assertions to be globally enabled by adding ASSERT=1 to the
build command line.  For example:

  make bin/8086100e.mrom ASSERT=1

Similarly, allow assertions to be globally disabled by adding ASSERT=0
to the build command line.  If no ASSERT=... is specified on the
build command line, then only objects mentioned in DEBUG=... will have
assertions enabled (as is currently the case).

Note than globally enabling assertions imposes a relatively heavy
runtime penalty, primarily due to the various sanity checks performed
by list_add(), list_for_each_entry(), etc.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Extend the DEBUG=... syntax to allow debug messages to be compiled in
but disabled by default.  For example:

  make bin/undionly.kpxe DEBUG=netdevice:3:1

would compile in the messages as for DEBUG=netdevice:3, but would set
the debug level mask so that only the DEBUG=netdevice:1 messages would
be displayed.

This allows for external code to selectively enable the additional
debug messages at runtime, without being overwhelmed by unwanted
initial noise.  For example, a developer of a new protocol may want to
temporarily enable tracing of all packets received: this can be done
by building with DEBUG=netdevice:3:1 and using

  // temporarily enable per-packet messages
  DBG_ENABLE_OBJECT ( netdevice, DBGLVL_EXTRA );
  ...
  // disable per-packet messages
  DBG_DISABLE_OBJECT ( netdevice, DBGLVL_EXTRA );

Note that unlike the usual DBG_ENABLE() and DBG_DISABLE() macros,
DBG_ENABLE_OBJECT() and DBG_DISABLE_OBJECT() will not be removed via
dead code elimination if debugging is disabled in the specified
object.  In particular, this means that using either of these macros
will always result in a symbol reference to the specified object.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The dependency on zlib seems to have been introduced in commit 3dd7ce1
("[efi] Allow building with non-system libbfd") as an indirect
requirement of either libbfd or libiberty when building on Mac OS X.
Since we no longer use either of these libraries, remove the
unnecessary link against zlib.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Parse the intermediate ELF file directly instead of using libbfd, in
order to allow for cross-compiled ELF objects.

As a side bonus, this eliminates libbfd as a build requirement.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

On some architectures (such as ARM) the "@" character is used as a
comment delimiter.  A section type argument such as "@progbits"
therefore becomes "%progbits".

This is further complicated by the fact that the "%" character has
special meaning for inline assembly when input or output operands are
used, in which cases "@progbits" becomes "%%progbits".

Allow the section type character(s) to be defined via Makefile
variables.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Some BIOS console redirection capabilities do not work well with the
colourised debug messages used by iPXE.  We already allow the range of
colours to be controlled via the DBGCOL=... build parameter.  Extend
this syntax to allow DBGCOL=0 to be used to mean "disable colours".

Requested-by: Wissam Shoukair <wissams@mellanox.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

The build system allows for additional drivers (or other objects) to
be specified using build targets such as

  make bin/intel--realtek.usb
  make bin/8086100e--8086100f.mrom

This currently fails if the base target is the "bin/ipxe.*" all-drivers
target, e.g.

  make bin/ipxe--acm.usb

Fix the build target parsing logic to allow additional drivers (or
other objects) to be included on top of the base all-drivers target.

This can be used to include USB network card drivers, which are not
yet included by default in the all-drivers build.

Reported-by: Andrew Sloma <asloma@lenovo.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

The name "vesafb" is intrinsically specific to a BIOS environment.
Generalise the build configuration option CONSOLE_VESAFB to
CONSOLE_FRAMEBUFFER, in preparation for adding EFI framebuffer
support.

Existing configurations using CONSOLE_VESAFB will continue to work.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Commit 7d36a1b ("[build] Explicitly link efilink against -liberty")
introduced a dependency on libiberty to cope with old versions of
libbfd.  This commit dates from 2008 and seems to apply only to what
are now extremely old versions of libbfd (prior to binutils 2.12).

There are systems (such as current Debian) which do not include
libiberty within the binutils packages.  On such systems, our build
dependency on libiberty represents a pointless hurdle.

Remove the explicit dependency on libiberty, hoping that there are no
modern systems where this will cause a problem.

Suggested-by: Ben Hildred <42656e@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Commit dc19e63 ("[build] Construct all-drivers list based on driver
class") accidentally excluded the USB bus drivers from the list of
files parsed in order to create PCI 3.0 device ID lists.

Fix by returning $(DRIVERS) to its previous definition as a list of
all driver files, and use only $(DRIVERS_ipxe) to contain the
filtered list containing only those drivers which we want to include
in the "all-drivers" build.

Reported-by: Mary-Ann Johnson <MaryAnn.Johnson@displaylink.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

The USB bus drivers (ehci.c and xhci.c) have PCI device ID tables and
hence PCI_ROM() lines, but should probably not be included in the
all-drivers build on this basis, since they do nothing useful unless a
USB network driver is also present.

Fix by constructing the all-drivers list based on the driver class
(i.e. the portion of the source path immediately after "drivers/").

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Inspired-by: Robin Smidsrød <robin@smidsrod.no>
Tested-by: Robin Smidsrød <robin@smidsrod.no>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

PCI v3.0 supports a "device list" which allows the ROM to claim
support for multiple PCI device IDs (but only a single vendor ID).
Add support for building such ROMs by scanning the build target
element list and incorporating any device IDs into the ROM's device
list header.  For example:

  make bin/8086153a--8086153b.mrom

would build a ROM claiming support for both 8086:153a and 8086:153b.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

LZMA provides significantly better compression (by ~15%) than the
current NRV2B algorithm.

We use a raw LZMA stream (aka LZMA1) to avoid the need for code to
parse the LZMA2 block headers.  We use parameters {lc=2,lp=0,pb=0} to
reduce the stack space required by the decompressor to acceptable
levels (around 8kB).  Using lc=3 or pb=2 would give marginally better
compression, but at the cost of substantially increasing the required
stack space.

The build process now requires the liblzma headers to be present on
the build system, since we do not include a copy of an LZMA compressor
within the iPXE source tree.  The decompressor is written from scratch
(based on XZ Embedded) and is entirely self-contained within the
iPXE source.

The branch-call-jump (BCJ) filter used to improve the compressibility
is specific to iPXE.  We choose not to use liblzma's built-in BCJ
filter since the algorithm is complex and undocumented.  Our BCJ
filter achieves approximately the same results (on typical iPXE
binaries) with a substantially simpler algorithm.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Hardened versions of gcc default to building position-independent
code, which breaks our i386 build.  Our build process therefore
detects such platforms and automatically adds "-fno-PIE -nopie" to the
gcc command line.

On x86_64, we choose to build position-independent code (in order to
reduce the final binary size and, in particular, the number of
relocations required for UEFI binaries).  The workaround therefore
breaks the build process for x86_64 binaries on such platforms.

Fix by moving the workaround to the i386-specific portion of the
Makefile.

Reported-by: Jan Kundrát <jkt@kde.org>
Debugged-by: Jan Kundrát <jkt@kde.org>
Debugged-by: Marin Hannache <git@mareo.fr>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

When building hvmloader for Xen tools the iPXE objects are also linked
into the binary.  Unfortunately the linker will place them in the
order found in the archive.  Since this order is random the resulting
hvmloader binary differs when it was built from identical sources but
on different build hosts.  To help with creating a reproducible binary
the elements in blib.a must simply be sorted before passing them to
$(AR).

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Add utility for constructing EFI fat binaries (dual 32/64-bit
binaries, usable only on Apple EFI systems).

This utility is not part of the standard build process.  To use it:

  make util/efifatbin bin-i386-efi/ipxe.efi bin-x86_64-efi/ipxe.efi

and then

  ./util/efifatbin bin-*-efi/ipxe.efi fat-ipxe.efi

Requested-by: Brandon Penglase <bpenglase-ipxe@spaceservices.net>
Tested-by: Brandon Penglase <bpenglase-ipxe@spaceservices.net>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Allow a straightforward "make clean" or "make veryclean" to apply to
all binary directories (using the shell pattern "bin{,-*}").
Individual binary directories can be cleaned using e.g.

  make bin clean
  make bin-x86_64-efi clean

Reported-by: Robin Smidsrød <robin@smidsrod.no>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

With extremely unlucky timing, it is possible to interrupt a build and
cause make to delete config/named.h (and possibly any local
configuration headers).

Mark config/named.h and all local configuration headers as .PRECIOUS
to prevent make from ever deleting them.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The build process has for a long time assumed that every ROM is a PCI
ROM, and will always include the PCI header and PCI-related
functionality (such as checking the PCI BIOS version, including the
PCI bus:dev.fn address within the ROM product name string, etc.).

While real ISA cards are no longer in use, some virtualisation
environments (notably VirtualBox) have support only for ISA ROMs.
This can cause problems: in particular, VirtualBox will call our
initialisation entry point with random garbage in %ax, which we then
treat as the PCI bus:dev.fn address of the autoboot device: this
generally prevents the default boot sequence from using any network
devices.

Create .isarom and .pcirom prefixes which can be used to explicitly
specify the type of ROM to be created.  (Note that the .mrom prefix
always implies a PCI ROM, since the .mrom mechanism relies on
reconfiguring PCI BARs.)

Make .rom a magic prefix which will automatically select the
appropriate PCI or ISA ROM prefix for ROMs defined via a PCI_ROM() or
ISA_ROM() macro.  To maintain backwards compatibility, we default to
building a PCI ROM for anything which is not directly derived from a
PCI_ROM() or ISA_ROM() macro (e.g. bin/intel.rom).

Add a selection of targets to "make everything" to ensure that the
(relatively obscure) ISA ROM build process is included within the
per-commit QA checks.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Allow named configurations to be specified via the CONFIG=... build
parameter.  For headers in config/*.h which support named
configurations, the following files will be included when building
with CONFIG=<name>:

  - config/defaults/<platform>.h (e.g. config/defaults/pcbios.h)

  - config/<header>.h

  - config/<name>/<header>.h (only if the directory config/<name> exists)

  - config/local/<header>.h (autocreated if necessary)

  - config/local/<name>/<header>.h (autocreated if necessary)

This mechanism allows for predefined named configurations to be
checked in to the source tree, as a directory config/<name> containing
all of the required header files.

The mechanism also allows for users to define multiple local
configurations, by creating header files in the directory
config/local/<name>.

Note that the config/*.h files which are used only to configure
internal iPXE APIs (e.g. config/ioapi.h) cannot be modified via a
named configuration.  This avoids rebuilding the entire iPXE codebase
whenever switching to a different named configuration.

Inspired-by: Robin Smidsrød <robin@smidsrod.no>
Tested-by: Robin Smidsrød <robin@smidsrod.no>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Commit 8290a95 ("[build] Expose build timestamp, build name, and
product names") introduced a regression in the build process which
resulted in broken final binaries which had names based on object
files (e.g. "undionly.kpxe" or "intel.rom") rather than on device IDs
(e.g. "8086100e.mrom").

The underlying problem is the -DOBJECT=<name> macro which is used to
generate the obj_<name> symbols used to select objects required for
the final binary.  The macro definition is derived from the initial
portion (up to the first dot) of the object being built.  In the case
of e.g. undionly.kpxe.version.o, this gives -DOBJECT=undionly.  This
results in undionly.kpxe.version.o claiming to be the "undionly"
object; the real "undionly" object will therefore never get dragged in
to the build.

Fix by renaming $(BIN)/%.version.o to $(BIN)/version.%.o, so that the
object is always built with -DOBJECT=version (as might be expected,
since it is built from core/version.c).

Final binaries which have names based on device IDs (such as
"8086100e.mrom") are not affected by this problem, since the object
name "8086100e" will not conflict with that of the underlying "intel"
object.

This problem was not detected by the per-commit smoke testing
procedure, which happens to use the binary bin/8086100e.mrom.

Reported-by: Christian Hesse <list@eworm.de>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Expose the build timestamp (measured in seconds since the Epoch) and
the build name (e.g. "rtl8139.rom" or "ipxe.efi"), and provide the
product name and product short name in a single centralised location.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

When multiple iPXE binaries are running concurrently (e.g. in the case
of undionly.kpxe using an underlying iPXE driver via the UNDI
interface) it would be helpful to be able to visually distinguish
debug messages from each binary.

Allow the range of debug colours used to be customised via the
DBGCOL=...  build parameter.  For example:

  # Restrict to colours 31-33 (red, green, yellow)
  make DBGCOL=31-33

  # Restrict to colours 34-36 (blue, magenta, cyan)
  make DBGCOL=34-36

Signed-off-by: Michael Brown <mcb30@ipxe.org>

If iPXE is used as a git submodule then the ../.git/index file will
not exist, and the build will fail.  Fix by checking that the git
index file exists before adding it as a build dependency.

Signed-off-by: Peter Lemenkov <lemenkov@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Commit 8540300 ("[build] Disable ccache for all relevant build
targets") attempted to generalise the rule for $(BIN)/version.o to
$(BIN)/version.% in order to apply the dependency to all relevant
build targets (debug objects, assembly listings, etc).

This generalisation appears to work for the ccache override
directives, but seems to cause make (at least, GNU make 4.0) to simply
ignore the dependency upon the git index.

Since version.c contains only some string constants, there is unlikely
to be a substantive need for its debug objects, assembly listings,
etc.  Restore the previous form of the dependency and accept that
hypothetical builds with e.g. DEBUG=version will not be handled
correctly.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Allow for an explicit debug level of zero, which will enable
assertions and profiling (i.e. anything controlled by NDEBUG) without
generating any debug messages.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Expand the concept of the X.509 cache to provide the functionality of
a certificate store.  Certificates in the store will be automatically
used to complete certificate chains where applicable.

The certificate store may be prepopulated at build time using the
CERT=... build command line option.  For example:

  make bin/ipxe.usb CERT=mycert1.crt,mycert2.crt

Certificates within the certificate store are not implicitly trusted;
the trust list is specified using TRUST=... as before.  For example:

  make bin/ipxe.usb CERT=root.crt TRUST=root.crt

This can be used to embed the full trusted root certificate within the
iPXE binary, which is potentially useful in an HTTPS-only environment
in which there is no HTTP server from which to automatically download
cross-signed certificates or other certificate chain fragments.

This usage of CERT= extends the existing use of CERT= to specify the
client certificate.  The client certificate is now identified
automatically by checking for a match against the private key.  For
example:

  make bin/ipxe.usb CERT=root.crt,client.crt TRUST=root.crt KEY=client.key

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Ensure that any generated files (such as DER forms of X.509
certificates) are rebuilt if the Makefile changes.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The build process currently attempts to disable ccache for files using
the .incbin directive, but the rule fails to apply to anything beyond
the simple object target.  Fix by applying to all relevant build
targets (including debug objects, assembly listings, and so on).

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Reported-by: Dewey Hylton <dewey@hyltown.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

OpenBSD 5.4 seems to generate dynamically linked binaries by default,
which breaks our build process.  Fix by forcing the linker to always
create static binaries.

Reported-by: Jiri B <jirib@devio.us>
Tested-by: Jiri B <jirib@devio.us>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Our use of --gc-sections causes the linker to discard the symbols
defined by FILE_LICENCE(), meaning that the resulting licence
determination is incomplete.

We must use the KEEP() directive in the linker script to force the
linker to not discard the licence symbols.  Using KEEP(*(COMMON))
would be undesirable, since there are some symbols in COMMON which we
may wish to discard.

Fix by placing symbols defined by PROVIDE_SYMBOL() (which is used by
FILE_LICENCE()) into a special ".provided" section, which we then mark
with KEEP().  All such symbols are zero-length, so there is no cost in
terms of the final binary size.

Since the symbols are no longer in COMMON, the linker will reject
symbols with the same name coming from multiple objects.  We therefore
append the object name to the licence symbol, to ensure that it is
unique.

Reported-by: Marin Hannache <git@mareo.fr>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

When the $(eval) function is available (in GNU make >= 3.80), we can
evaluate many of the dynamically-generated Makefile rules directly.
This avoids generating a few hundred Makefile fragments in the
filesystem, and so speeds up the build process.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Linker table entries must be non-static in order to avoid being
completely optimised away by some versions of gcc.  Use -Wno-decl to
prevent sparse from warning about these, since the alternative would
be to litter the code with otherwise unnecessary "extern"
declarations.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

Signed-off-by: Michael Brown <mcb30@ipxe.org>

sparse seems to have problems finding compiler.h when specified as
"-include compiler.h"; one possible explanation is that it ignores the
include path.  Fix by using "-include include/compiler.h".

Signed-off-by: Michael Brown <mcb30@ipxe.org>

The version number string is currently updated only if version.o
happens to be rebuilt due to changes in its dependencies.  Add a
dependency upon the git index, so that the version number is updated
after any checkout.

Signed-off-by: Thomas Miletich <thomas.miletich@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>