Replacing doxygen with mrdocs - #984

cc @alandefreitas

Hi Daniel, thanks for the cc, and glad to see simdutf trying this. simdutf is exactly the sort of codebase Mr.Docs is meant for.

I'm actually working on a Doxygen-to-Mr.Docs migration guide at this very moment and reworking the configuration reference at the moment. The categories, the examples, and the migration-notes page are all getting refactored in the next week or two, so if a particular option reads confusingly today, that page is likely to be in much better shape soon. Please leave questions or feedback or open an issue on mrdocs and I'll jump in.

A few notes from reading your mrdocs.yml:

• Some of your exclusions probably want implementation-defined or see-below instead of exclude-symbols. Your mrdocs_nav.py mentioned the issue: simdutf::detail types leak into the output because they appear in public-API signatures, and exclude-symbols drops the symbol entirely, leaving any signature that returns or takes one with a dangling reference. implementation-defined keeps the symbol in the corpus so cross-references resolve, but renders it as /* implementation-defined */ everywhere it appears and gives it no page. That's what you want for simdutf::detail::**, since that's the leakage mrdocs_nav.py is working around. see-below is the same idea applied to classes whose synopsis should be hidden but whose brief and description should still appear. The synopsis renders as class X { /* see-below */ }; and the members are dropped. simdutf::internal::**, simdutf::scalar::**, and simdutf::tables::** can stay on exclude-symbols if none of them appear in public-API signatures.
• On wrapper.html.hbs: most of what you're overriding is the inline CSS block. If you'd prefer, the stylesheets option appends external CSS files to the head on top of the defaults, so the addon directory shrinks to just the parts of the layout that aren't styling. The wrapper override can stay for structural changes, but the CSS doesn't need to live inside it.
• scripts/mrdocs_nav.py is well-built, and the missing sidebar nav in the html generator is something Mr.Docs should fix in the generator itself. As an alternative, it's worth mentioning global-namespace-index turns the global-namespace page into a project-wide index of top-level symbols when combined with multipage mode. Much rougher than what your tag-file post-processor produces, but it's there out of the box. The shape of your script is a good reference for what the built-in version should eventually look like.
• If there's a lot of reference documentation missing, another tip is you can enable the warn-as-error option temporarily from the command line and put that in a loop with an AI agent until there are no Mr.Docs errors. This gives you a preliminary version; you just need to review it once.

Thanks!

@alandefreitas Thanks for the feedback. I think that I addressed some of your concerns in my latest commit.

I think the generated docs look pretty. But how does it work with overloads?
In particular, if I go to for instance https://lemire.me/simdutf/mrdocs/simdutf/binary_to_base64.html only the pointer+size overload is there, the span overload is missing.

is it possible to get the other overloads visible as well?

Hi :)

MrDocs definitely creates pages for overload sets and for the individual functions (i.e. https://lemire.me/simdutf/mrdocs/simdutf/base64_valid-01.html)

But I don't know which functions in the library are overloaded and what choices the script that generates the index makes regarding them.

@pauldreik I have updated the script so that it generates C++23 docs, I am uploading the new files to https://lemire.me/simdutf/mrdocs/

@alandefreitas @pauldreik To be clear, the reason the overloads did not appear is that I was building with the default C++ version. Some of our overloads require a specific C++ version.

Here is an example:
https://lemire.me/simdutf/mrdocs/simdutf/binary_to_base64.html

there is only one overload (the base one, with ptr+size). the span overload is not visible.

I think this is important, because the span overload should be preferred at least by users having access to C++20 .

@pauldreik The reason you still found this link is that mrdocs does not delete the html files from previous runs. I have added a rm command to the script.

You still have it...

https://lemire.me/simdutf/mrdocs/simdutf/binary_to_base64-0c.html

but it is now part of a larger definition:

https://lemire.me/simdutf/mrdocs/simdutf/binary_to_base64-0c.html

The downside of my approach here is that it relies on the development version of mrdocs, so it could fail in the future.

Also, maybe I could convince @alandefreitas to issue a release soon.

@alandefreitas @pauldreik I can't use the latest release. I am not sure I remember why. It is missing some feature I needed.

Yes. The leftover happened because the page names changed. When the corpus had a single binary_to_base64, its page was binary_to_base64.html, no suffix. Now three symbols share that name (the overload set plus the two overloads), so the pages are binary_to_base64-0c.html, -05, and -01, and nothing generates the unsuffixed file anymore. It just stayed behind until your rm removed it.

There's still one leftover on the site though: the sidebar. The mrdocs-nav.js published right now links binary_to_base64 to the unsuffixed page, which 404s after the cleanup, and it still shows the old single arglist. The reference.tag.xml next to it is the new one (it points to -0c), so it looks like scripts/mrdocs_nav.py didn't run again after the clean rebuild, or the deploy picked up an old copy. Rerunning it should fix the links. One heads-up when you do: in the new tagfile, an overloaded function comes through as its overload set, a compound with kind "class", not as a namespace function member. So binary_to_base64 will show up under "Types" instead of "Functions" unless the script special-cases that.

And yes, a release is coming soon, right after some documentation updates we're finishing. So many people build straight from develop that we ended up with fewer and fewer releases. I've updated master to match develop in the meantime, in case pinning that helps.

@pauldreik We could adopt this.

Hi! I did a full rewrite of the MrDocs docs (https://www.mrdocs.com/docs/). The one I was talking about in the first comment I left here. You might be interested in seeing it. It’s more goal-oriented, it has examples with previews for everything, the reference pages are exhaustive, and there are more pages on categories of commands and options. This makes it easier to understand what's going on with some of the issues we had here, because they weren't properly explained before.

@lemire can you please regenerate the docs? I clicked a random page on the left hand side, and it got me a 404.
specifically this one: https://lemire.me/simdutf/mrdocs/simdutf/utf8_length_from_utf16_with_replacement.html

@pauldreik I think that you have a staled (cached) page.

The link should be

https://lemire.me/simdutf/mrdocs/simdutf/utf8_length_from_utf16_with_replacement-0e.html