[riot-devel] Updates to the build system - modules definition

Gaëtan Harter gaetan.harter at fu-berlin.de
Fri Jan 5 17:41:44 CET 2018


Hi all,


I want to give my opinion on the discussions that happened here in 
Berlin regarding the build system evolution.
And also, choose the direction for next development steps.



# Laze presentation #

Kaspar presented his new tool that replaces the current Make based build 
system based on a custom python software reading the configuration and 
generating `ninja` files to builds.
It uses YAML description for modules, application, and toolchain elements.


GOOD

* The YAML description is a good direction into what I need as a 
"descriptive language" for the modules.
* Using a meta build-system, removes the need to parse all the 
configuration from scratch each times an application is build which is a 
static time consumption for CI.
  It could be done in make but may be more cumbersome to write and maintain.
* Refined module build configuration to only show the needed flags and 
dependencies, it will help a lot for caching.
* I love python.

BAD

* Python is not a build system language, so not designed to do 'targets' 
generation but more executing commands after the other.
* Introducing our own custom build language framework instead of using 
an existing one, so a non standard tool.
* The build system implementation (how to parse yaml files, generate 
build rules), is not separated from the laze tool itself, so hard to 
have the implementation stored in the repository while having a generic 
tool installed globally.
* Many argument are based on speed, but the speed improvement are not 
all a result of using 'laze' only, but cleaning up concepts in general.


# Criticism of the current make build system #

However, I agree that our current build system has a lot of flaws that I 
would like to see solved.

* Its necessary to run `make clean` all the time because there are 
changes that do not correctly trigger rebuild, so you are never sure
* The module definition/configuration is not structured and descriptive 
so is problematic to provide finer granularity module building, all 
modules see all CFLAGS, INCLUDES, and included modules MODULE_ macros.
     * Its now impossible to cache results between applications as they 
are done with different config.
* Not all dependencies are explicitly described, so hard to understand 
the optional dependencies, the dependencies on interface, dependencies 
depending on configuration. This would also help refining module build 
configuration.
* Our current build system only takes care of building one application 
for one board. For the CI and maintainers, when building several 
targets, they must re-execute make from scratch for each configuration.
* And the initial  tool checking, is it linux, modules discovery, should 
be done every time. And this really adds up on many targets builds
     * It can only be solved correctly by first having structured module 
description as you can parse it once and apply it to different 
configurations
* There is no support for building in a common build directory so it 
takes a lot of space and also rebuilds multiple time the same thing.
     * It currently makes no sense, as module have different 
configurations, but after fixing the previous issues, one built module 
configuration could be shared between different apps, saving space and 
compile time.


# Next steps together #

To go forward and use what he developed as a possible solution, I 
proposed to start a documentation phase, on both his new system + YAML 
module format, and on what and how things are currently done in the 
build system.
Adding user and developer documentation for the current system would 
fill the lack of documentation and also help describing all the 
currently existing features, show the problems, find forgotten requirements.
And documenting his description format and his tool could show what are 
the benefits of his choices and also the tradeoffs.


One of my question is, is it interesting to integrate some changes in 
the current make system, like some early module description, even if 
replaced later by another build system ?

I will open issues on github to follow the next steps.

Cheers,
Gaëtan









On 01.12.2017 10:49, Kaspar Schleiser wrote:
> Hi Matthias,
>
> On 12/01/2017 10:29 AM, Matthias Waehlisch wrote:
>>> When are you gonna take a look at my ninja-based build system? It
>>> solves 1-3 quite nicely, and more. You could save a lot of time.
>>>
>>    Pointer?
> https://github.com/kaspar030/RIOT/tree/add_laze_buildfiles
>
> Don't expect a nice out-of-the-box experience (yet).
>
> Kaspar
> _______________________________________________
> devel mailing list
> devel at riot-os.org
> https://lists.riot-os.org/mailman/listinfo/devel



More information about the devel mailing list