Mercurial > p > roundup > code

annotate .github/workflows/build-xapian.yml @ 8491:520075b29474

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
feat: support justhtml parsing library to convert email to plain text justhtml is an pure python, fast, HTML5 compliant parser. It is now an option for converting html only emails to plain text. Its output format differs slightly from dehtml or beautifulsoup. Mostly by removing extra blank lines. dehtml.py: Using the stream parser of justhtml. Unable to get the full document parser to successfully strip script and style blocks. If I can fix this and use the standard parser, I can in theory generate markdown from the DOM tree generated by justhtml. Updated test case to include inline elements that should not cause a line break when they are encountered. Running dehtml as: `python roundup/dehtml.py foo.html` will load foo.html and parse it using all available parsers. configuration.py: justhtml is available as an option. docs: updated CHANGES.txt, doc/tracker_config.txt added beautifulsoup and justhtml to the optional software section of doc/installtion.txt. test_mailgw.py, .github/workflows/ci-test Updated tests and install justhtml as part of CI.
author John Rouillard <rouilj@ieee.org>
date Sun, 14 Dec 2025 22:40:46 -0500
parents 4e0944649af7
children 951db0950174
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 name: build-xapian
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 on:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 push:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6 # skip if github.ref is 'refs/heads/maint-1.6'
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 # aka github.ref_name of 'maint-1.6'
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8 # see https://github.com/orgs/community/discussions/26253
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 # for mechanism to control matrix based on branch
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 branches: [ "*", '!maint-1.6' ]
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 workflow_dispatch:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 inputs:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 debug_enabled:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14 type: boolean
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 description: 'Run the build with tmate debugging enabled (https://github.com/marketplace/actions/debugging-with-tmate)'
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 required: false
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 default: false
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18 # GITHUB_TOKEN only has read repo context.
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19 permissions:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20 contents: read
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22 concurrency:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 group: ${{ github.workflow }}-${{ github.ref }}
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 cancel-in-progress: true
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 jobs:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 test:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 name: build xapian
8156
131c962fb6af try 24.04 issue2551338 - xapian doesn't build in CI for 3.13 python
John Rouillard <rouilj@ieee.org>
parents: 7963
diff changeset
29 runs-on: ubuntu-24.04
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 env:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 # get colorized pytest output even without a controlling tty
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 PYTEST_ADDOPTS: "--color=yes"
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 # OS: ${{ matrix.os }}
7951
9009d489bf5a test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7950
diff changeset
35 PYTHON_VERSION: 3.13
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 steps:
7950
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
38 # Checkout the latest code from the repo
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
39 - name: Checkout source
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
40 # example directives:
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
41 # disable step
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
42 # if: {{ false }}
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
43 # continue running if step fails
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
44 # continue-on-error: true
8489
4e0944649af7 chore: update actions/checkout from 6.0.0 to 6.1.1 pull74
John Rouillard <rouilj@ieee.org>
parents: 8483
diff changeset
45 uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
7950
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
46
29730a09e882 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7949
diff changeset
47 # Setup version of Python to use
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48 - name: Set Up Python 3.13
8483
7201316bd6e8 chore: bump actions/setup-python from 6.0.0 to 6.1.0
John Rouillard <rouilj@ieee.org>
parents: 8479
diff changeset
49 uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
50 with:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 python-version: 3.13
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 allow-prereleases: true
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 cache: 'pip'
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 - name: Install build tools - setuptools
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 run: pip install setuptools
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 # Display the Python version being used
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 - name: Display Python and key module versions
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 run: |
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 python -c "import sys; print('python version: ', sys.version)"
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 python -c "import sqlite3; print('sqlite version: ', sqlite3.sqlite_version)"
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 python -c "import setuptools; print('setuptools version: ', setuptools.__version__);"
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65 - name: Update pip
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66 run: python -m pip install --upgrade pip
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 # https://github.com/mxschmitt/action-tmate
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69 # allow remote ssh into the CI container. I need this to debug
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 # some xfail cases
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 - name: Setup tmate session
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 uses: mxschmitt/action-tmate@v3
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
7956
9166e0100e01 test: 1.4.18 get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7955
diff changeset
74 timeout-minutes: 60
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 with:
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 limit-access-to-actor: true
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 - name: Install xapian
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79 run: |
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 set -xv
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 sudo apt-get install libxapian-dev
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 # Sphinx required to build the xapian python bindings. Use 1.8.5 on
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
83 # older python and newest on newer.
7963
05ce472d372b test: 1.4.25 get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7956
diff changeset
84 pip install sphinx
8157
7c9d209b598a set xapian version 1.4.22 - xapian doesn't build in CI for 3.13 python
John Rouillard <rouilj@ieee.org>
parents: 8156
diff changeset
85 XAPIAN_VER="1.4.22"; echo $XAPIAN_VER;
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
86 cd /tmp
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
87 curl -s -O https://oligarchy.co.uk/xapian/$XAPIAN_VER/xapian-bindings-$XAPIAN_VER.tar.xz
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
88 tar -Jxvf xapian-bindings-$XAPIAN_VER.tar.xz
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89 cd xapian-bindings-$XAPIAN_VER/
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 # edit the configure script.
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 # distutils.sysconfig.get_config_vars('SO') doesn't work for
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92 # 3.11 or newer.
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
93 # Change distutils.sysconfig... to just sysconfig and SO
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
94 # to EXT_SUFFIX to get valid value.
7963
05ce472d372b test: 1.4.25 get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents: 7956
diff changeset
95 # DISABLED use their script
7949
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
96 if [[ $PYTHON_VERSION == "X."* ]]; then
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
97 cp configure configure.FCS;
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 sed -i \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
99 -e '/PYTHON3_SO=/s/distutils\.//g' \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
100 -e '/PYTHON3_SO=/s/"SO"/"EXT_SUFFIX"/g' \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
101 -e '/PYTHON3_CACHE_TAG=/s/imp;print(imp.get_tag())/sys;print(sys.implementation.cache_tag)/' \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
102 -e '/PYTHON3_CACHE_OPT1_EXT=/s/imp\.get_tag()/sys.implementation.cache_tag/g' \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103 -e '/PYTHON3_CACHE_OPT1_EXT=/s/imp\b/importlib/g' \
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 configure;
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 diff -u configure.FCS configure || true;
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
106 fi
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
107 ./configure --prefix=$VIRTUAL_ENV --with-python3 --disable-documentation
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
108 make && sudo make install
5cc2d0001723 test: get xapian working under github actions and python 3.13
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
109 python -c 'import xapian'
Roundup Issue Tracker: http://roundup-tracker.org/