ipxe/contrib/compressor/algorithm.doc

The  compressor achieves  an  average compression  rate of 60%  of the
original size which is on par with "gzip". It seems that you cannot do
much better for compressing  compiled  binaries.  This means that  the
break even  point  for using compressed  images is   reached, once the
uncompressed size approaches 1.5kB. We  can stuff more than 12kB  into
an 8kB EPROM and more than 25kB into an 16kB EPROM.   As there is only
32kB of RAM  for both the uncompressed  image  and its BSS  area, this
means that 32kB EPROMs will hardly ever be required.

The compression  algorithm uses a  4kB  ring buffer  for buffering the
uncompressed data. Before   compression starts,  the  ring buffer   is
filled  with spaces (ASCII  character  0x20).  The algorithm tries  to
find repeated  input sequences of a  maximum length of  60 bytes.  All
256 different input  bytes  plus the 58 (60   minus a threshold of  2)
possible  repeat lengths form a set  of 314 symbols. These symbols are
adaptively Huffman encoded.  The  algorithm starts out with a Huffmann
tree  that  assigns equal code lengths    to each of  the  314 symbols
(slightly favoring the repeat  symbols over symbols for regular  input
characters), but  it will be changed whenever  the frequency of any of
the symbols  changes. Frequency counts are  kept in 16bit  words until
the total number of compressed codes totals 2^15.  Then, all frequency
counts will be halfed (rounding to the bigger number).  For unrepeated
characters (symbols 0..255) the Huffman code  is written to the output
stream.  For repeated characters the  Huffmann code, which denotes the
length of the repeated character sequence, is written out and then the
index in the ring buffer is computed.   From this index, the algorithm
computes  the offset   relative to  the current  index  into  the ring
buffer. Thus,  for typical input data,  one would expect that short to
medium range offsets are more frequent  than extremely short or medium
range to long range offsets. Thus the  12bit (for a 4kB buffer) offset
value  is statically Huffman encoded  using a precomputed Huffman tree
that favors  those  offset  values    that  are deemed to   be    more
frequent. The  Huffman encoded offset  is  written to the output  data
stream,  directly  following the code  that   determines the length of
repeated characters.

This algorithm, as implemented in the  C example code, looks very good
and  its operating parameters are   already well optimized. This  also
explains   why  it achieves     compression ratios    comparable  with
"gzip". Depending on the input data, it sometimes excells considerably
beyond what "gzip -9" does, but this  phenomenon does not appear to be
typical. There are some flaws with  the algorithm, such as the limited
buffer  sizes, the  adaptive  Huffman tree  which takes  very  long to
change, if    the input  characters  experience   a sudden   change in
distribution, and the static Huffman   tree for encoding offsets  into
the  buffer.   The slow  changes of   the  adaptive  Huffman  tree are
partially counteracted by  artifically keeping  a 16bit precision  for
the frequency counts, but  this does not  come into play until 32kB of
compressed data is output, so  it does not  have any impact on our use
for "etherboot", because  the BOOT Prom  does not support uncompressed
data of more then 32kB (c.f. doc/spec.doc).

Nonetheless,  these problems  do  not  seem  to affect  compression of
compiled  programs very much.  Mixing  object code with English  text,
would not work too  well though, and  the algorithm should be reset in
between. Actually, we  might  gain a little  improvement, if  text and
data   segments    were compressed  individually,    but   I have  not
experimented with this option, yet.