diff doc/upgrading.txt @ 7860:8b31893f5930

issue2551115/issue2551282 - utf8mb4 support in roundup Fix issues with utf8 support in Roundup. By default using: utf8mb4 charset utf8mb4_unicode_ci collation (case insensitive) utf8mb4_0900_ci collation (case sensitive) which are settable from config.ini. Sadly I couldn't come up with a way to mange these from one parameter. Doing a compatibility lookup table would have increased the maintenance burden and have me chasing MySQL changes. So I opted for the easy path and have the admins (with more MySQL experience) make the choices. Conversion directions added to upgrading.txt. I don't have any good testing for this. I was able to generate utf8/utf8mb3 tables and load a little data and convert. However this is a poor substitute for a conversion on a working tracker 8-(.
author John Rouillard <rouilj@ieee.org>
date Sat, 06 Apr 2024 22:47:25 -0400
parents 0fe2b9f6e19f
children f47b186a2ad9
line wrap: on
line diff
--- a/doc/upgrading.txt	Sat Apr 06 20:37:45 2024 -0400
+++ b/doc/upgrading.txt	Sat Apr 06 22:47:25 2024 -0400
@@ -112,6 +112,128 @@
 
 This will insert the bad API login rate limiting settings.
 
+Update MySQL character set/collations (required)
+------------------------------------------------
+
+issue2551282_ and issue2551115_ discuss issues with MySQL's utf8
+support. MySQL has variations on utf8 character support. This
+version of Roundup expects to use utf8mb4 which is a version of
+utf8 that covers all characters, not just the ones in the basic
+multilingual plane. Previous versions of Roundup used latin1 or
+utf8mb3 (also known as just utf8). Newer versions of MySQL are
+supposed to make utf8mb4 and not utf8mb3 the default.
+
+To convert your database, you need to have MySQL 8.0.11 or newer
+(April 2018) and a mysql client.
+
+.. warning::
+
+   This conversion can damage your database. Back up your
+   database using mysqldump or other tools. Preferably on a quiet
+   database.  Verify that your database can be restored (or at
+   least look up directions for restoring it). This is very
+   important.
+
+We suggest shutting down Roundup's interfaces:
+
+  * web
+  * email
+  * cron jobs that use Python or roundup-admin
+
+then make your backup.
+
+Then connect to your mysql instance using ``mysql`` with the
+information in ``config.ini``. If your tracker's ``config.ini``
+includes::
+
+  name = roundupdb
+  host = localhost
+  user = roundupuser
+  password = rounduppw
+
+you would run some version of::
+
+  mysql -u roundupuser --host localhost -p roundupdb
+
+and supply ``rounduppw`` when prompted.
+
+With the Roundup database quiet, convert the character set for the
+database and then for all the tables. To convert the tables you
+need a list of them. To get this run::
+
+    mysql -sN -u roundupuser --host localhost -p \
+       -e 'show tables;' roundupdb > /tmp/tracker.tables
+
+The ``-sN`` removes line drawing characters and column headers
+from the output. For each table ``<t>`` in the file, run::
+
+ ALTER TABLE `<t>` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+You can automate this conversion using sed::
+
+    sed -e 's/^/ALTER TABLE `/' \
+     -e 's/$/` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;/'\
+     /tmp/tracker.tables> /tmp/tracker.tables.sql
+
+The backticks "`" are required as some of the table names became
+MySQL reserved words during Roundup's lifetime.
+
+Inspect ``tracker.tables.sql`` to see if all the lines look
+correct. If so then we can start the conversion.
+
+First convert the character set for the database by running::
+
+  mysql -u roundupuser --host localhost -p roundupdb
+
+Then at the ``mysql>`` prompt run::
+
+  ALTER DATABASE roundupdb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+you should see: ``Query OK, 1 row affected (0.01 sec)``.
+
+Now to modify all the tables run:
+
+  \. /tmp/tracker.tables.sql
+
+You will see output similar to::
+
+  Query OK, 5 rows affected (0.01 sec)
+  Records: 5  Duplicates: 0  Warnings: 0
+
+for each table. The rows/records will depend on the number of
+entries in the table. This can take a while.
+
+Once you have successfully completed this, copy your tracker's
+config.ini to a backup file. Edit ``config.ini`` to use the defaults:
+
+ * mysql_charset = utf8mb4
+ * mysql_collation = utf8mb4_unicode_ci
+ * mysql_binary_collation = utf8mb4_0900_bin
+
+Also look for a ``~/.my.cnf`` for the roundup user and make sure
+that the settings for character set (charset) are utf8mb4 compatible.
+
+To test, run ``roundup-admin -i tracker_home`` and display an
+issue designator: e.g. ``display issue10``. Check that the text
+fields are properly displayed (e.g. title). Start the web
+interface and browse some issues. Again, check that the text
+fields display correctly, that the history at the bottom of the
+issues displays correctly and if you are using the default full
+text search, make sure that that works.
+
+If this works, bring email cron jobs etc. back online.
+
+If this fails, take down the web interface, restore the database
+from backup, restore the old config.ini. Then test again and
+reach out to the mailing list for help.
+
+We can use assistance in getting these directions corrected or
+enhanced. The core Roundup developers don't use MySQL for their
+production workloads so we count on users to help us with this.
+
+.. _issue2551282: https://issues.roundup-tracker.org/issue2551282
+.. _issue2551115: https://issues.roundup-tracker.org/issue2551115
+
 Disable performance improvement for wsgi mode (optional)
 --------------------------------------------------------
 

Roundup Issue Tracker: http://roundup-tracker.org/