Status and future of Xerox and other FST tools
Presently the Giella infrastructure supports three fst technologies in parallel:
- Hfst
- Xerox
- Foma
Each of them have their strengths and weaknesses, summarised as follows:
Hfst
This compiler is the default choice for the ./configure setup, i.e. it is
the compiler you use if you do not specify what compiler you want.
It is a source-code compatible clone of the Xerox tools (below), based on OpenFST, but with multiple backends (Foma, SFST, OpenFST).
Strengths:
- replicates 99% of the functionality of the Xerox tools
- is actively maintained
- is almost as fast in lookup as Xerox when using the optimised lookup format
- supports weighted fst’s
- open source
Weaknesses:
- is slow in compiling fst’s compared to Xerox
Xerox
Xerox is at present (spring 2021) the compiler used for cgi-bin services and online dictionaries.
Strengths:
- fast in lookup
- fast in compiling source code
Weaknesses:
- no support for weights
- closed source
- abandoned by its main developer Lauri Karttunen (he is retired, and his effort to open-source it was blocked by Xerox lawyers)
- For some languages (e.g. sms), Xerox is very slow.
Source code access
Even though the source code is not released, it is possible to get a license to the source code of the c-fsm library (documented here) by requesting a license for the XLE page. Information and relevant links can be found at the bottom of the project page.
Foma
A source-code and command interface compatible clone of the Xerox xfst tool, developed and maintained by Måns Huldén. Is open source.
Strengths:
- fast in lookup
- fast in compiling source code
- actively maintained
Weaknesses:
- no support for twolc
- limited support for weighted fst’s
How to cope with this…
… ie the lack of future for Xerox, the lack of twolc in Foma and the lack of speed in Hfst.
Today’s dual strategy
- use the slow hfst for all applications demanding open source and/or weighting
- use the fast xfst for developing and for some applications
Having two compilers for different appliations becomes increasingly difficult.
The future
… is dependent upon
- whether hfst will become faster
- whether foma will include support for twolc
- whether it will be possible to license some version of c-fst that will produce the command line tools we need
We should now migrate all applications to hfst.