@@ -19,8 +19,9 @@
 set -o errexit
 
 if [ $# -ne 4 ]; then
-  printf "Usage: %s lang_dir LM lexicon out_dir\n" `basename $0`
-  echo "  Convert ARPA-format language models to FSTs.";
+  echo "Usage: $0 <lang_dir> <arpa-LM> <lexicon> <out_dir>"
+  echo "E.g.: $0 data/lang data/local/lm/foo.kn.gz data/local/dict/lexicon.txt data/lang_test"
+  echo "Convert ARPA-format language models to FSTs.";
   exit 1;
 fi
 
@@ -34,7 +35,7 @@ mkdir -p $out_dir
 
 echo "Converting '$lm' to FST"
 
-for f in phones.txt words.txt L.fst L_disambig.fst phones/ oov.int oov.txt; do
+for f in phones.txt words.txt topo L.fst L_disambig.fst phones/ oov.int oov.txt; do
   cp -r $lang_dir/$f $out_dir
 done
 
 
@@ -29,7 +29,7 @@ if [ $# -ne 4 ] && [ $# -ne 3 ]; then
   echo "E.g.: utils/format_lm_sri.sh data/lang data/local/lm/foo.kn.gz data/local/dict/lexicon.txt data/lang_test"
   echo "Converts ARPA-format language models to FSTs. Change the LM vocabulary using SRILM."
   echo "Note: if you want to just convert ARPA LMs to FSTs, there is a simpler way to do this"
-  echo "that doesn't require SRILM: see examples in egs/wsj/s5/local/wsj_format_local_lms.sh"
+  echo "that doesn't require SRILM: see utils/format_lm.sh"
   echo "options:"
   echo " --help                 # print this message and exit"
   echo " --srilm-opts STRING      # options to pass to SRILM tools (default: '$srilm_opts')"
 
@@ -810,44 +810,32 @@ www.openfst.org and do the FST tutorial so that you understand the basics of fin
 state transducers.  (Note that language models would be represented as finite state
 acceptors, or FSAs, which can be considered as a special case of finite state transducers).
 
-In the WSJ setup (like most setups), we use a statistical language model.  The script <DFN>local/wsj_format_data.sh</DFN>
-deals with converting the ARPA-format language models supplied with the WSJ database, into
-an OpenFst format.  Some of the key commands from that script are:
-\verbatim
-  gunzip -c $lmdir/lm_${lm_suffix}.arpa.gz | \
-   utils/find_arpa_oovs.pl $test/words.txt  > $tmpdir/oovs_${lm_suffix}.txt
- ...
-  gunzip -c $lmdir/lm_${lm_suffix}.arpa.gz | \
-    grep -v '<s> <s>' | \
-    grep -v '</s> <s>' | \
-    grep -v '</s> </s>' | \
-    arpa2fst - | fstprint | \
-    utils/remove_oovs.pl $tmpdir/oovs_${lm_suffix}.txt | \
-    utils/eps2disambig.pl | utils/s2eps.pl | fstcompile --isymbols=$test/words.txt \
-      --osymbols=$test/words.txt  --keep_isymbols=false --keep_osymbols=false | \
-     fstrmepsilon > $test/G.fst
-\endverbatim
-Here, the variable <DFN>\$test</DFN> would be something like <DFN>data/lang_test_tg</DFN>.
-The most important of these commands is <DFN>arpa2fst</DFN> which is a Kaldi program.
-This program turns the ARPA-format language model into a Weight Finite State Transducer
-(actually, an acceptor).
-The <DFN>grep</DFN> commands remove certain "invalid" n-grams from the language model, which
-should not have been there in the first place.  The program <DFN>remove_oovs.pl</DFN> removes
-N-grams containing words not in our vocabulary (which would cause a crash in <DFN>fstcompile</DFN>); <DFN>eps2disambig.pl</DFN> turns the
-<DFN>\<eps\></DFN> (\f$\epsilon\f$) symbols on backoff arcs into a special symbol <DFN>#0</DFN> which
-we use to make the grammar determinizable; see \ref graph_disambig.  If you don't know
-what a "backoff arc" is, see the literature on backoff N-grams, for instance Goodman's "A bit of progress in
-language modeling", and the paper by Mohri we referenced above.  The command <DFN>s2eps.pl</DFN> turns the
-sentence-begin and sentence-end symbols <DFN>\<s\></DFN> and <DFN>\</s\></DFN> into
-epsilon (<DFN>\<eps\></DFN>), which means "no symbol"; <DFN>fstcompile</DFN> is an OpenFst
-command which turns the
-FST from text form into OpenFst binary form; and <DFN>fstrmepsilon</DFN> is also
-an OpenFst command, which removes the small number of <DFN>\<eps\></DFN> symbols from the
-FST, which were converted from <DFN>\<s\></DFN> and <DFN>\</s\></DFN>.
+The script <DFN>utils/format_lm.sh</DFN> deals with converting the ARPA-format language
+models into an OpenFst format. Here is the usage messages of that script: 
+\verbatim
+Usage: utils/format_lm.sh <lang_dir> <arpa-LM> <lexicon> <out_dir>
+E.g.: utils/format_lm.sh data/lang data/local/lm/foo.kn.gz data/local/dict/lexicon.txt data/lang_test
+Convert ARPA-format language models to FSTs.
+\endverbatim
+Some of the key commands from that script are:
+\verbatim
+gunzip -c $lm \
+  | arpa2fst --disambig-symbol=#0 \
+             --read-symbol-table=$out_dir/words.txt - $out_dir/G.fst
+\endverbatim
+This Kaldi program, <DFN>arpa2fst</DFN>, turns the ARPA-format language model
+into a Weight Finite State Transducer (actually, an acceptor).
 
 A popular toolkit for building language models is SRILM.  Various language
 modeling toolkits are used in the Kaldi example scripts.  SRILM is the best
 documented and most fully featured, and we generally recommend it (its only
-drawback is that it don't have the most free licence).
+drawback is that it don't have the most free licence). Here is the usage
+messages of <DFN>utils/format_lm_sri.sh</DFN>
+
+\verbatim
+Usage: utils/format_lm_sri.sh [options] <lang-dir> <arpa-LM> <out-dir>
+E.g.: utils/format_lm_sri.sh data/lang data/local/lm/foo.kn.gz data/lang_test
+Converts ARPA-format language models to FSTs. Change the LM vocabulary using SRILM.
+\endverbatim
 
 */
@@ -199,19 +199,9 @@ The steps we have to do aside from just running arpa2fst are as follows:
 
 A slightly simplified version of the actual script that does this is as follows:
 \verbatim
-  gunzip -c data_prep/lm.arpa.gz | \
-     scripts/find_arpa_oovs.pl data/words.txt  > data/oovs.txt
-
-  gunzip -c data_prep/lm.arpa.gz | \
-    grep -v '<s> <s>' | \
-    grep -v '<s> </s>' | \
-    grep -v '</s> </s>' | \
-     ../src/lm/arpa2fst - | fstprint | \
-    scripts/remove_oovs.pl data/oovs.txt | \
-    scripts/eps2disambig.pl | \
-     fstcompile --isymbols=data/words.txt --osymbols=data/words.txt \
-     --keep_isymbols=false --keep_osymbols=false > data/G.fst
-  fstisstochastic data/G.fst
+gunzip -c data_prep/lm.arpa.gz | \
+  arpa2fst --disambig-symbol=#0 \
+             --read-symbol-table=data/words.txt - data/G.fst
 \endverbatim
 The last command (fstisstochastic) is a diagnostic step (see \ref fst_algo_stochastic).  
 In one typical example, it prints out the numbers:
 
@@ -339,31 +339,16 @@ ngram-count -text train.txt -order 3 -limit-vocab -vocab wordlist -unk \
 Now that we have the ARPA format language model trained, we have to compile it
 into WFST format. Let's first define the following variables
 \verbatim
+dict_dir=data/local/dict                # The dict directory provided by the online-nnet2 models
 lm=srilm.o3g.kn.gz                      # ARPA format LM you just built.
 lang=data/lang                          # Old lang directory provided by the online-nnet2 models
 lang_own=data/lang_own                  # New lang directory we are going to create, which contains the new language model
-lang_own_tmp=data/local/lang_own_tmp/   # Temporary directory.
 \endverbatim
 
 Given the above variables, we can compile an ARPA format language model into
 WFST format using the following commands
 \verbatim
-mkdir -p $lang_own_tmp
-mkdir -p $lang_own
-cp -r $lang/* $lang_own
-gunzip -c $lm | utils/find_arpa_oovs.pl $lang_own/words.txt \
-  > $lang_own_tmp/oovs.txt || exit 1
-gunzip -c $lm | \
-  grep -v '<s> <s>' | \
-  grep -v '</s> <s>' | \
-  grep -v '</s> </s>' | \
-  arpa2fst - | fstprint | \
-  utils/remove_oovs.pl $lang_own_tmp/oovs.txt | \
-  utils/eps2disambig.pl | utils/s2eps.pl | \
-  fstcompile --isymbols=$lang_own/words.txt --osymbols=$lang_own/words.txt  \
-  --keep_isymbols=false --keep_osymbols=false | \
-  fstrmepsilon | fstarcsort --sort_type=ilabel > $lang_own/G.fst
-utils/validate_lang.pl --skip-determinization-check $lang_own || exit 1;
+utils/format_lm.sh $lang $lm $dict_dir/lexicon.txt $lang_own
 \endverbatim
 
 Now, we can compile the decoding graph with the new language model, using the
@@ -408,6 +393,7 @@ just created, and "<SPOKEN_NOISE>" is the word symbol that represents
 out-of-vocabulary words in the lexicon, we can generate the new lang directory
 with the updated lexicon using the following command
 \verbatim
+lang_own_tmp=data/local/lang_own_tmp/   # Temporary directory.
 utils/prepare_lang.sh \
   --phone-symbol-table $lang/phones.txt \
   $dict_own "<SPOKEN_NOISE>" $lang_own_tmp $lang_own