[riot-devel] Updates to the build system - modules definition

Gaëtan Harter gaetan.harter at fu-berlin.de
Thu Nov 30 16:32:40 CET 2017

Hi Thomas, Francisco, all,

I will try to summarize with three aspects formatted as problems.

Just for clarity, in the following text, use the words 'description', 
'informations', 'configuration', 'dependencies' to describe all the 
meta-information that can be attached to a module in addition to its 
source code.

I also am in demand of other issues that you have and that should be 
taken into account if the current system is refactored.
So I invite you to talk about your case too, I want to start discussions 
before working on it.

Also I welcome all remarks regarding my problems and the way I want to 
address them.

      1. Configuration is not documented

As Francisco is saying, about configuring an application and so its modules.
The problem, for me, is that its hard to know the possible 
configurations of a module and the impact of the configuration/dependencies.


When building an application, I need to:

  * describe what I want to use in my application, which usually means
    saying which module/package I want
  * configure the application and the modules, by using either specific
    variables `DEVELHELP`, `OUTPUT` or manually setting CFLAGS
  * Specify requirements to work only on boards with this hardware support.

  * But it also means understanding the impact of adding a specific
    module / setting one configuration

Right now the impact is tedious to find. A dependency pulled by one 
module can have an impact on another module code, on its API, on all 
And modules configuration, is often only described into the doxygen or 
the source code: what CFLAGS should I put to configure the radio channel ?

I would like to clearly write down these public configurations and 
dependencies relations.

      2. Information is not readable

The module information cannot be presented and re-interpreted to show on 
a graphical interface.
It is not machine readable, and the same way of defining things has 
multiple meaning.


I would need to have parseable files usable by an automated system, 
which is currently not the case.

For my needs, it would be necessary to refine the current information to 
make it more precise.
The 'dependencies' and configurations now used in the build system can 
be ambiguous to present because they have different meaning.

  * Some optional dependencies have different impact, they:
      o can change the source code behavior
      o can change the module API
      o can affect every modules
  * The same goes for configuration, is the configuration affecting the
    module API ?

      3. Modules definition is scattered but in RIOT global files

The meta-information on a module is not grouped together with the module 
but written in 3-4 files global to the RIOT repository.
This means reading different files to get a complete view on a module 
but also that it is not possible to use external modules trees without 
modifying the RIOT repository.


I would like to make modules self contained. I think it would help 
developers as it improves readability and documentation by having only 
one place to understand a module.
Also, external modules would be defined in the same way as internal ones 
by just adding a tree of modules to load.

        Bonus. Unnecessary globally visible configuration

A bonus aspect that I would like to tackle in the long term when the 
updates would be in place.

Right now all CFLAGS/INCLUDES are global to all modules.
This means a different version of the `core` module is build if another 
high level module is included or has its configuration changed.

Having more informations on modules dependencies and configuration scope 
could help refine this.


Having a determined context would makes tests more reliable, because we 
could know which configuration has been tested and which not.
It would require explicit build system configuration to make a 
configuration affect another module.
Solving this could allow doing build caching at a the build system level 
when build with common local context.
I am also interested in reproducible builds and so being able to 
describe what has been build with the minimal information required and 
verifying the produced output.

With these issues in mind, I propose to add parseable module meta-data 
definitions in a file in each module directory to first replace the 
existing information and then extend it to add more precise ones.
How to do in practice it is still to be defined and discussed.

Gaëtan - cladmi

On 11/27/2017 04:39 PM, Francisco Javier Acosta Padilla wrote:
> Hi,
> I think the problem to solve is that RIOT is very difficult to configure.
> While working with other IoT operating systems, one can notice that 
> RIOT is very hardly configurable, unless we are very familiar with the 
> build system and specific RIOT CFLAGS and special key-words 
> Besides, I have no knowledge of a tool that can parse RIOT makefiles 
> and give an insight about module usage, dependencies, features, etc.
> Thus IMHO a way to make, first, RIOT more *friendly* configurable it’s 
> a good and necessary feature. This would also allow to eventually 
> build a dependency tree in a more human-readable fashion, as well as 
> give some help to get different product lines according to the 
> selected features.
> However, I’d say that *deep* modifications to the current build system 
> can lead to instability or buggy builds. I’m not against any 
> improvement to the build system, but maybe as a first approach some 
> additions like files in any more descriptive (parseable) language can 
> help to describe RIOT in a higher level, without affecting at all the 
> underlying system. Then, with the knowledge obtained thanks to such a 
> description, we can imagine to generate makefiles or make “make” calls 
> with the specific RIOT CFLAGS, USEMODULE, etc.
> My 2c.
> Cheers,
> -- 
> Francisco Javier Acosta Padilla
> Research Engineer at INRIA Saclay
> On 24 November 2017 at 19:16:29, Thomas C. Schmidt 
> (t.schmidt at haw-hamburg.de <mailto:t.schmidt at haw-hamburg.de>) wrote:
>> Hi Dan, Gaetan,
>> I wonder, what problems are we trying to solve?
>> Maybe we can clearly enumerate them first so that we can check later,
>> whether an improved solution really solves these problems.
>> Cheers,
>> Thomas
>> On 24/11/2017 16:47, Daniel Petry wrote:
>> > Hi Gaetan
>> >
>> >
>> >> I would like to introduce some packaging concepts around RIOT.
>> >> To consider modules like well defined distribution packages and 
>> not only
>> >> source files added to an application firmware.
>> >
>> > From what you're saying, I understand that the aim is to make modules
>> > completely self-contained with respect to build definitions, and make
>> > those definitions much more human readable. In this way, an application
>> > developer can go to only one place to find out information regarding a
>> > module's build (dependency information, etc.), and a module developer
>> > only has to write/change the module directory Makefile and can do so
>> > with declarative language. Also, the information can be easily 
>> parseable
>> > for eg a user interface.
>> >
>> > So the changes you're proposing are:
>> >
>> > 1) Move build information concerning a particular module into that
>> > module's Makefile
>> > 2) Make the module makefiles able to be written with purely declarative
>> > language
>> > 3) Retain backwards compatibility with the current build system
>> >
>> > Is this correct?
>> >
>> > I think 2) would be great for user friendliness, and 3) is a 
>> no-brainer.
>> > 1) is a bigger topic as there are a number of different ways in which
>> > dependencies are manifested for example, so I think this can be a point
>> > for further discussion... do you have any particular examples?
>> >
>> >
>> > Dan.
>> >
>> > _______________________________________________
>> > devel mailing list
>> > devel at riot-os.org
>> > https://lists.riot-os.org/mailman/listinfo/devel
>> >
>> -- 
>> Prof. Dr. Thomas C. Schmidt
>> ° Hamburg University of Applied Sciences Berliner Tor 7 °
>> ° Dept. Informatik, Internet Technologies Group 20099 Hamburg, Germany °
>> ° http://www.haw-hamburg.de/inet Fon: +49-40-42875-8452 °
>> ° http://www.informatik.haw-hamburg.de/~schmidt Fax: +49-40-42875-8409 °
>> _______________________________________________
>> devel mailing list
>> devel at riot-os.org
>> https://lists.riot-os.org/mailman/listinfo/devel
> _______________________________________________
> devel mailing list
> devel at riot-os.org
> https://lists.riot-os.org/mailman/listinfo/devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riot-os.org/pipermail/devel/attachments/20171130/1e0fd1e1/attachment-0001.html>

More information about the devel mailing list