Libdoc: Don't convert Markdown to HTML when generating spec files by default #5808

RF 7.5 enhanced Libdoc so that documentation can be converted in Markdown (#5304). The main motivation was making it possible to use a commonly used documentation format instead of Robot's custom format, not that widely used reStructuredText or plain HTML. When compared to HTML, Markdown has a benefit that it takes less space and is easier to process programmatically, especially when using AI tools. Markdown is thus a good format in machine readable JSON or XML spec files.

How Libdoc, by default, handles spec file documentation depends on the spec file format:

• When generating JSON spec files, docs are converted to HTML.
• When generating XML spec files with the generic XML extension, the original documentation format is preserved.
• When generating XML spec files with the custom .libspec extension (Libdoc: Support *.libspec extension when reading library information from spec files #3491), docs are converted to HTML.

The reason that JSON and LIBSPEC files use HTML by default is that it made more sense than using Robot's custom format that external tools did not understand. With XML specs the format is preserved, because changing it would have been a backwards incompatible change. Regardless the spec format, it is possible to use --specdocformat HTML to convert docs to HTML and --specdocformat RAW to preserve the original format.

With Markdown the situation has now changed to that Markdown (i.e. the original source format) makes more sense than HTML. When we added Markdown support, we thus should have handled it so that it is not, by default, converted to HTML regardless the spec format. Due to backwards compatibility reasons it's a bit questionable to change it afterwards, but I believe the benefits are nevertheless bigger and we should do that in RF 7.6.

It can be argued that we should wait for RF 8.0 before making this change, but I don't think this is such a big change that it needed to wait for a major version. Only users who'd be affected are early Markdown adopters and I would assume that most of them prefer preserving the format anyway. The longer we have Markdown support available before making this change, the more there are users that are affected. Mitigating the backwards compatibility issue is also easy:

• If you prefer preserving Markdown, use --specdocformat RAW with RF 7.5. It won't be needed anymore after RF 7.6 is released, but using it causes no harm either.
• If you prefer converting Markdown to HTML, take --specdocformat HTML into use. You can do it already now, but you need to do it latest before migrating to RF 7.6.

It could also be argued that this should be changed already in RF 7.5.1, but I don't think backwards incompatible changes are acceptable in bug fix releases except in very special cases. I believe we should document that the behavior will change in the User Guide, though, and also mention that as a deprecation in RF 7.5.1 release notes. That requires a separate issue.