Word‐based text compression

Abstract

The development of efficient algorithms to support arithmetic coding has meant that powerful models of text can now be used for data compression. Here the implementation of models based on recognizing and recording words is considered. Move‐to‐the‐front and several variable‐order Markov models have been tested with a number of different data structures, and first the decisions that went into the implementations are discussed and then experimental results are given that show English text being represented in under 2‐2 bits per character. Moreover the programs run at speeds comparable to other compression techniques, and are suited for practical use. Copyright © 1989 John Wiley & Sons, Ltd