Mercurial > p > roundup > code

annotate roundup/cgi/accept_language.py @ 5108:67fad01d2009

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
issue2550653: xapian search, stemming is not working This is a partial fix for the issue. It does make stemming work (so searching for silent will also return docs with silently in them). However to do this we need to lowercase the text so the porter stemmer will work. This means capitalization is not preserved. Tests in test/test_indexer for xapian backend all pass. David Wolever (wolever) did the work.
author John Rouillard <rouilj@ieee.org>
date Mon, 27 Jun 2016 22:10:45 -0400
parents 74476eaac38a
children b00cd44fea16
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3426
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
1 """Parse the Accept-Language header as defined in RFC2616.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
2
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
3 See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
4 for details. This module should follow the spec.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
5 Author: Hernan M. Foffani (hfoffani@gmail.com)
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
6 Some use samples:
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
7
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
8 >>> parse("da, en-gb;q=0.8, en;q=0.7")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
9 ['da', 'en_gb', 'en']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
10 >>> parse("en;q=0.2, fr;q=1")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
11 ['fr', 'en']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
12 >>> parse("zn; q = 0.2 ,pt-br;q =1")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
13 ['pt_br', 'zn']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
14 >>> parse("es-AR")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
15 ['es_AR']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
16 >>> parse("es-es-cat")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
17 ['es_es_cat']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
18 >>> parse("")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
19 []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
20 >>> parse(None)
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
21 []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
22 >>> parse(" ")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
23 []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
24 >>> parse("en,")
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
25 ['en']
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
26 """
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
27
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
28 import re
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
29 import heapq
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
30
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
31 # regexp for languange-range search
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
32 nqlre = "([A-Za-z]+[-[A-Za-z]+]*)$"
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
33 # regexp for languange-range search with quality value
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
34 qlre = "([A-Za-z]+[-[A-Za-z]+]*);q=([\d\.]+)"
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
35 # both
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
36 lre = re.compile(nqlre + "|" + qlre)
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
37
4362
74476eaac38a more modernisation
Richard Jones <richard@users.sourceforge.net>
parents: 3426
diff changeset
38 ascii = ''.join([chr(x) for x in range(256)])
3426
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
39 whitespace = ' \t\n\r\v\f'
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
40
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
41 def parse(language_header):
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
42 """parse(string_with_accept_header_content) -> languages list"""
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
43
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
44 if language_header is None: return []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
45
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
46 # strip whitespaces.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
47 lh = language_header.translate(ascii, whitespace)
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
48
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
49 # if nothing, return
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
50 if lh == "": return []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
51
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
52 # split by commas and parse the quality values.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
53 pls = [lre.findall(x) for x in lh.split(',')]
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
54
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
55 # drop uncomformant
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
56 qls = [x[0] for x in pls if len(x) > 0]
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
57
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
58 # use a heap queue to sort by quality values.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
59 # the value of each item is 1.0 complement.
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
60 pq = []
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
61 for l in qls:
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
62 if l[0] != '':
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
63 heapq.heappush(pq, (0.0, l[0]))
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
64 else:
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
65 heapq.heappush(pq, (1.0-float(l[2]), l[1]))
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
66
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
67 # get the languages ordered by quality
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
68 # and replace - by _
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
69 return [x[1].replace('-','_') for x in pq]
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
70
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
71 if __name__ == "__main__":
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
72 import doctest
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
73 doctest.testmod()
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
74
52f89836d05b Parse the Accept-Language header as defined in RFC2616.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
diff changeset
75 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/