[riot-notifications] [RIOT-OS/RIOT] [WIP] doc: Sphinx for documentation / second attempt. (#11459)

Juan I Carrano notifications at github.com
Fri Apr 26 19:41:26 CEST 2019

### Contribution description

This is a second shot at #9369.

[This is how it looks like.](https://riot-os.readthedocs.io/) The changelog looks ugly, but I already know how to fix it.

 <summary>Click here for the original PR description</summary>

## Documentation in RIOT

Right now, RIOT is (partly) documented using Doxygen. While it is a great tool, it has its limitations. The main problem with Doxygen is that it is a tool designed for API reference. Documentation is much more than API reference. Ideally we would want to have a "RIOT handbook" that walks the user through the different parts of the system, and guides the developer who wants to improve RIOT, alongside the API reference.

Doxygen does an excellent job at capturing and extracting the low-level details, but has no support for higher-level concepts (like modules, packages or custom constructs). It does not have the flexibility to support them either. In general, it is very oriented towards documenting files and C++ (or C, or Java) constructs. For C the problem is worse because several concepts which are used in practice (like objects and interfaces) do not have a formal representation in thus language and are thus hard to document.

Regarding the flexibility of doxygen: it is not easy to customize the output. The HTML output does not (easily) support full text search. The only way to organize the docs is with groups, whose meaning is not very well defined, and one ends up with the same entity appearing in multiple pages.

Last, if a modules has really good documentation, it will probably be too much to have it as a comment in a header file.

## [Sphinx](http://www.sphinx-doc.org/)

Check out the preview here: https://riot-os.readthedocs.io/ . It's still missing everything.

### Benefits

* Full text search
* Customizable and themeable output
* Support for automatic documentation using Doxygen and ~[Breathe](http://breathe.readthedocs.io/)~ [antidox](https://github.com/riot-appstore/antidox).
* Support for custom ["domains"](http://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html). We could define a RIOT domain whose roles are, for example "package", "module", "board", etc. 
* If necessary, we can make our own extensions.
* We can publish on [Read the Docs](https://readthedocs.org/).
* We can structure the docs like we want.
* Supports markdown, so we don't need to rewrite what we have. There are [reasons](http://ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/) for writing new docs in reStructuredText, though.

### Downsides

* Requires a more manual involvement for writing and organizing the docs.
* The build process is slower an more complex.
* The integration between Doxygen and Sphinx is not perfect (though it can be improved)
* It is more noticeable if we are missing something (maybe this is good). Doxygen, in contrast, is specially good at extracting loads of content.

### Example

The Zephyr project is using Sphinx+Breathe+Doxygen. Check it out [here](http://docs.zephyrproject.org/index.html).

### Procedure

1. Doxygen generates XML output.
2. ~`breathe-apidoc` reads the XML and generates reStructuredText documents with indices (e.g. `doxygenclass` directives)~ [I need to figure out this step]
3. Sphinx (with the ~breathe~ antidox extension) reads hand written MD and RST documents, plus the ones generated in step 2. When a `doxygen*` directive is found, the XML database is read and the appropriate document nodes are generated.  


The last time I tried to use breathe+doxygen+readthedocs only to find out that the _massive_ size of our _tiny_ OS was too much for breathe. Not only was it going over the allocated time on readthedocs, even running it on my office machine turned it into a 700W heater.

I took a detour and wrote [my own Sphinx extension](https://pypi.org/project/antidox/). With it the build can be completed in just under the 15 minutes time limit of RTD (if we want to use that, there's always the possibility of building it ourselved).

I'm aware of the general sentiment towards having homebrew code as dependencies. For that reason I tried to be extra tidy in developing the extension and to document everything. I also made it customizable via XLST and Sphinx events so if somebody does not like the output, it can be changed.

Being more customizable will also allow us to work around doxygen bugs like #10427 , #10815 , etc.

### Testing procedure

Like with anything involving new python packages, it is recommended that you create a virtualenv first.

$ python -m venv ~/sphinxriot
$ . ~/sphinxriot/bin/activate

Now you can build the docs:

$ cd doc/sphinx
$ pip install -r requirements.txt
$ make

Open `_build/html/index.html` (in Linux, you can also type `make view`).

### Issues/PRs references

There are 6370 warnings. There are lots repeated entities, some that are missing, and other stuff, as expected from a [mindless list of automatically generated stuff](https://varnish-cache.org/docs/trunk/phk/sphinx.html).

Closes #9369.
You can view, comment on, or merge this pull request online at:


-- Commit Summary --

  * doxygen/Makefile: enable XML generation.
  * doc: Initial Sphinx support.

-- File Changes --

    M .gitignore (1)
    M doc/doxygen/Makefile (4)
    A doc/sphinx/.gitignore (2)
    A doc/sphinx/Makefile (53)
    A doc/sphinx/_static/.keep (0)
    A doc/sphinx/conf.py (219)
    A doc/sphinx/gen_stubs.py (108)
    A doc/sphinx/index.rst (28)
    A doc/sphinx/requirements.txt (1)

-- Patch Links --


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riot-os.org/pipermail/notifications/attachments/20190426/c6d8852d/attachment.html>

More information about the notifications mailing list