[Buildroot] CMake and VERBOSE variable

[Cédric Marie]

Hi,

CMake turns on verbose mode if VERBOSE environment variable is defined
and not empty. VERBOSE=0, VERBOSE=1, VERBOSE=whatever all enable
verbose mode.
See for example: http://www.cmake.org/Bug/view.php?id=3378

If VERBOSE is defined but empty, a "light" verbose mode is enabled. It
consists in printing this information every time xxx.c file needs to be
recompiled:
Dependee "xxx.c" is newer than depender "xxx.c.o".

NB: I am aware that these light verbose messages don't appear at first
build. It is only disturbing when rebuilding packages when a source
file is modified.

This light verbose mode is enabled in Buildroot, and I suppose this is
not deliberate. This is caused by VERBOSE being exported in root
Makefile (line 231), even if not set:
export SHELL CONFIG_SHELL quiet Q KBUILD_VERBOSE VERBOSE

NB: All other variables in that list seems to always be not empty.

There is another environment variable, CMAKE_NO_VERBOSE, that can be
set (=1) to disable these "dependee" messages, even if VERBOSE is
defined and empty.
It does not disable the full verbose mode. VERBOSE=1 can still be used,
but the "dependee" messages will not be showned in this full (not so
full...) verbose mode.

To disable this unexpected light verbose mode, the first solution could
consist in adding CMAKE_NO_VERBOSE=1 in $(2)_MAKE_ENV in pkg-cmake.mk.

This would make the fix specific to CMake, which makes sense for a CMake
specific problem :)

In fact I have not really made up my mind whether the "real" problem is
in CMake or in Buildroot. On the one hand, CMake behaviour is strange
(checking an empty variable), and on the other hand, Buildroot should
not export this variable, even if this is not supposed to be harmful.

In the end, I think the right solution would be not to export VERBOSE
if not set (remove it from the export list, and export it when
defined).

What do you think about it? The fact that it is defined empty does not
seem to be useful anywhere, but I might have missed something...

Regards,

[Arnout Vandecappelle]

Post by Cédric Marie
Hi,
CMake turns on verbose mode if VERBOSE environment variable is defined
and not empty. VERBOSE=0, VERBOSE=1, VERBOSE=whatever all enable
verbose mode.
See for example: http://www.cmake.org/Bug/view.php?id=3378
If VERBOSE is defined but empty, a "light" verbose mode is enabled. It
consists in printing this information every time xxx.c file needs to be
Dependee "xxx.c" is newer than depender "xxx.c.o".
NB: I am aware that these light verbose messages don't appear at first
build. It is only disturbing when rebuilding packages when a source
file is modified.
This light verbose mode is enabled in Buildroot, and I suppose this is
not deliberate. This is caused by VERBOSE being exported in root
export SHELL CONFIG_SHELL quiet Q KBUILD_VERBOSE VERBOSE

The reason to export is, is to pass it to sub-makes that make use of the same
mechanism, e.g. the kernel.

However, there's something to be said for exporting it only if V=1.

Post by Cédric Marie
NB: All other variables in that list seems to always be not empty.
There is another environment variable, CMAKE_NO_VERBOSE, that can be
set (=1) to disable these "dependee" messages, even if VERBOSE is
defined and empty.
It does not disable the full verbose mode. VERBOSE=1 can still be used,
but the "dependee" messages will not be showned in this full (not so
full...) verbose mode.
To disable this unexpected light verbose mode, the first solution could
consist in adding CMAKE_NO_VERBOSE=1 in $(2)_MAKE_ENV in pkg-cmake.mk.
This would make the fix specific to CMake, which makes sense for a CMake
specific problem :)

The thing is, the working of CMAKE_NO_VERBOSE is not intuitive either (because
it can still be combined with VERBOSE=1).

So I'm more in favour of not exporting VERBOSE if empty.

Regards,
Arnout

Post by Cédric Marie
In fact I have not really made up my mind whether the "real" problem is
in CMake or in Buildroot. On the one hand, CMake behaviour is strange
(checking an empty variable), and on the other hand, Buildroot should
not export this variable, even if this is not supposed to be harmful.
In the end, I think the right solution would be not to export VERBOSE
if not set (remove it from the export list, and export it when
defined).
What do you think about it? The fact that it is defined empty does not
seem to be useful anywhere, but I might have missed something...
Regards,

[Cédric Marie]

Hi,

Post by Arnout Vandecappelle
The reason to export is, is to pass it to sub-makes that make use of the same
mechanism, e.g. the kernel.
However, there's something to be said for exporting it only if V=1.

Is it OK to export it when defined? i.e.:

ifndef VERBOSE
export VERBOSE = 1
endif

This is the local fix I'm using.

Post by Arnout Vandecappelle

Post by Cédric Marie
NB: All other variables in that list seems to always be not empty.

Replying to myself: No, most of them are also exported empty, but it is
alright. Only VERBOSE will have a special meaning when defined empty.

Post by Arnout Vandecappelle
The thing is, the working of CMAKE_NO_VERBOSE is not intuitive either (because
it can still be combined with VERBOSE=1).

The different kinds of verbose modes seem to be neither intuitive nor
documented...


Thank you,

[Arnout Vandecappelle]

Post by Cédric Marie
Hi,

Post by Arnout Vandecappelle
The reason to export is, is to pass it to sub-makes that make use of the same
mechanism, e.g. the kernel.
However, there's something to be said for exporting it only if V=1.

ifndef VERBOSE
export VERBOSE = 1
endif

Actually, no. It should also be exported if the user had set it like so:

make V=1 VERBOSE=3

Therefore, it should become (with full context otherwise other readers will not
understand why there's an ifndef)

ifeq ($(KBUILD_VERBOSE),1)
quiet =
Q =
ifndef VERBOSE
VERBOSE = 1
endif
+export VERBOSE
else
quiet = quiet_
Q = @
endif


However, on second thought: why do we set VERBOSE at all? It was introduced in
2007 by Bernhard Reutner-Fischer with the explanation:

set and export VERBOSE if V= was requested

Not much of an explanation...

A quick review of a few typical packages (linux, busybox, uClibc, u-boot,
autotools-baed) only turns up uClibc that has a not-very-well documented use of
VERBOSE. So perhaps we should just revert Bernhard's patch and completely remove
VERBOSE.

Peter?

Post by Cédric Marie
This is the local fix I'm using.

Post by Arnout Vandecappelle

Post by Cédric Marie
NB: All other variables in that list seems to always be not empty.

Replying to myself: No, most of them are also exported empty, but it is alright.
Only VERBOSE will have a special meaning when defined empty.

Well, there's no way to be sure of that...

In fact, the export of quiet Q KBUILD_VERBOSE comes directly from the kernel
(another commit from Bernhard, log "adjust infrastructure for new kconfig"). But
of course, in the kernel the build system has complete control over what will be
called with this stuff in the environment. For us that's a lot more complicated.
So I think we should get rid of all these exports, and leave it up to individual
packages to do that. Only V is something that is really generally used I think.


Regards,
Arnout

Post by Cédric Marie

Post by Arnout Vandecappelle
The thing is, the working of CMAKE_NO_VERBOSE is not intuitive either (because
it can still be combined with VERBOSE=1).

The different kinds of verbose modes seem to be neither intuitive nor documented...
Thank you,

[Cédric Marie]

Hi,

Post by Arnout Vandecappelle
A quick review of a few typical packages (linux, busybox, uClibc, u-boot,
autotools-baed) only turns up uClibc that has a not-very-well
documented use of
VERBOSE. So perhaps we should just revert Bernhard's patch and
completely remove
VERBOSE.
Peter?
In fact, the export of quiet Q KBUILD_VERBOSE comes directly from the kernel
(another commit from Bernhard, log "adjust infrastructure for new kconfig"). But
of course, in the kernel the build system has complete control over what will be
called with this stuff in the environment. For us that's a lot more complicated.
So I think we should get rid of all these exports, and leave it up to individual
packages to do that. Only V is something that is really generally used I think.

Indeed, all these settings are done the same way in the root Makefile of
the kernel.
Since KBUILD_VERBOSE doesn't seem to be used anywhere else in Buildroot
(not even in support/kconfig), I also tend to think that this could be
removed.

What should we do next?
I'm not familiar enough with all these verbose settings, and also with
the kernel, to be able to check that we're not breaking anything when
removing them.
Should I first send a patch that moves the export in the right place
that you have suggested?
Or wait for a cleaning of all these exports?

Regards,

[Arnout Vandecappelle]

Hi,

Post by Arnout Vandecappelle
A quick review of a few typical packages (linux, busybox, uClibc, u-boot,
autotools-baed) only turns up uClibc that has a not-very-well documented use of
VERBOSE. So perhaps we should just revert Bernhard's patch and completely remove
VERBOSE.
Peter?
In fact, the export of quiet Q KBUILD_VERBOSE comes directly from the kernel
(another commit from Bernhard, log "adjust infrastructure for new kconfig"). But
of course, in the kernel the build system has complete control over what will be
called with this stuff in the environment. For us that's a lot more complicated.
So I think we should get rid of all these exports, and leave it up to individual
packages to do that. Only V is something that is really generally used I think.

Indeed, all these settings are done the same way in the root Makefile of the
kernel.
Since KBUILD_VERBOSE doesn't seem to be used anywhere else in Buildroot (not
even in support/kconfig), I also tend to think that this could be removed.
What should we do next?
I'm not familiar enough with all these verbose settings, and also with the
kernel, to be able to check that we're not breaking anything when removing them.
Should I first send a patch that moves the export in the right place that you
have suggested?
Or wait for a cleaning of all these exports?

I propose to do it in two patches:

- a first patch that removes the export of VERBOSE, with the commit log
explaining how it breaks cmake projects;

- a second patch that removes the export of all the rest, with the commit log
explaining that there is no reason to export it and that it was just inherited
from Kbuild.

Can you do that?

Regards,
Arnout

[Cédric Marie]

Hi,

Post by Arnout Vandecappelle
- a first patch that removes the export of VERBOSE, with the commit log
explaining how it breaks cmake projects;
- a second patch that removes the export of all the rest, with the commit log
explaining that there is no reason to export it and that it was just inherited
from Kbuild.
Can you do that?

OK, I will do that.

Regards,

[Samuel Martin]

Hi all,

On Mon, Jun 22, 2015 at 11:47 AM, Cédric Marie

Post by Cédric Marie
Hi,

Post by Arnout Vandecappelle
- a first patch that removes the export of VERBOSE, with the commit log
explaining how it breaks cmake projects;

Does this mean that the VERBOSE env. var. should now be handled by the
cmake-infra?

Post by Cédric Marie

Post by Arnout Vandecappelle
- a second patch that removes the export of all the rest, with the commit log
explaining that there is no reason to export it and that it was just inherited
from Kbuild.
Can you do that?

OK, I will do that.
Regards,

Regards,

[Cédric Marie]

Post by Samuel Martin

Post by Arnout Vandecappelle
- a first patch that removes the export of VERBOSE, with the commit log
explaining how it breaks cmake projects;

Does this mean that the VERBOSE env. var. should now be handled by the
cmake-infra?

No, you can still build with "make VERBOSE=1" and CMake will take it
into account.
There is nothing to change in cmake-infra.

The patch only prevents from systematically exporting VERBOSE when
empty, which has a specific meaning for CMake - intentional or not, I
don't know.

[Samuel Martin]

All,

On Mon, Jun 22, 2015 at 1:26 PM, Cédric Marie

Post by Samuel Martin

Post by Arnout Vandecappelle
- a first patch that removes the export of VERBOSE, with the commit log
explaining how it breaks cmake projects;

Does this mean that the VERBOSE env. var. should now be handled by the
cmake-infra?

No, you can still build with "make VERBOSE=1" and CMake will take it into
account.

Not really convenient :(
AFAICS, VERBOSE is not an official env. var. altering Buildroot's
behavior, unlike V [1].

If VERBOSE should only be set for cmake-based package, in addition to
globally unexport it, I'd prefer:
- either fix the cmake infra to keep the same behavior from the user
point of view;
- or update the doc.

I prefer the 1st option.

There is nothing to change in cmake-infra.
The patch only prevents from systematically exporting VERBOSE when empty,
which has a specific meaning for CMake - intentional or not, I don't know.

I think it is intentional (given [2] and the number of occurences of
"\<VERBOSE\>" in the cmake code base ;-]).


[1] http://nightly.buildroot.org/#make-tips
[2] http://www.cmake.org/Wiki/CMake_FAQ#Is_there_an_option_to_produce_more_.27verbose.27_compiling.3F

Regards,

[Cédric Marie]

Post by Samuel Martin

No, you can still build with "make VERBOSE=1" and CMake will take it into
account.

Not really convenient :(
AFAICS, VERBOSE is not an official env. var. altering Buildroot's
behavior, unlike V [1].
If VERBOSE should only be set for cmake-based package, in addition to
- either fix the cmake infra to keep the same behavior from the user
point of view;
- or update the doc.
I prefer the 1st option.

I guess you're right. I was not aware of that.

Please note that the first patch will not remove the export of VERBOSE.
It will export it only if V is set.
The patch proposed by Arnout was:

ifeq ($(KBUILD_VERBOSE),1)
quiet =
Q =
ifndef VERBOSE
VERBOSE = 1
endif
+export VERBOSE
else
quiet = quiet_
Q = @
endif

Which means that you can still enable CMake verbose mode with "make
V=1".


In the second patch, we are planning to remove: quiet Q KBUILD_VERBOSE
VERBOSE
It would be possible to keep VERBOSE (and export it only if V is set),
but I agree that it would make more sense to turn V into VERBOSE in
cmake-infra, since it is CMake specific, and CMake does not understand
V.

But then, what about autotools?
V is supposed to be taken into account by autotools, but I can't see no
difference with V=0 or V=1, even when forcing it in <pkg>_BUILD_CMDS.
(Checked with gdb for example).

Post by Samuel Martin
I think it is intentional (given [2] and the number of occurences of
"\<VERBOSE\>" in the cmake code base ;-]).

What I suggested to be non-intentional is the empty VERBOSE case, which
produces only the "dependee" messages.
I have already contacted CMake dev team about that, but I couldn't get a
clear answer for the empty VERBOSE case.
The discussion is here, I have pointed out the source code that was
responsible for that behaviour (see the last message of the discussion):
http://www.cmake.org/pipermail/cmake/2014-February/056943.html

[Cédric Marie]

Hi!

I'm getting back to working on this old subject (it was discussed in
June).
I have proposed a first patch to fix the CMake specific problem.
See:
http://git.buildroot.net/buildroot/commit/?id=307029867b0384446cd74b232b45a7b4f40cf0d1
VERBOSE is not exported anymore, unless V=1 in the command line.

It was suggested that I should provide a second patch, to address the
problem globally.
I have a local patch for that, but I need some feedback before providing
it.

The idea is to remove exported variables that come from the kernel and
are useless in our context (KBUILD_VERBOSE and quiet), to remove
VERBOSE, and to manage the verbosity in a specific way in each pkg-infra
(VERBOSE=1 is understood by CMake, but might not be understood by
another build system).

When V=1, we should export Q = (empty) and a new variable BR2_VERBOSE =
1.
When V=0, we should export Q = @ and BR2_VERBOSE = (empty).

Q behaviour is not modified.

BR2_VERBOSE is taken into account in pkg-infra.
* In pkg-cmake.mk:
When BR2_VERBOSE=1, I add VERBOSE=1 in <PKG>_BUILD_CMDS
* In pkg-autotools.mk:
I don't know what to do. VERBOSE=1 has no effect, V=0/1 has no effect,
--enable-silent-rules has no effect.

NB: The only verbosity setting that has an effect on autotools is "make
-s", but it is different from "make V=0/1", and it is already taken into
account by Buildroot (QUIET variable in root Makefile).

Globally, there is no change:
* V=1 involves VERBOSE=1 for CMake.
* V=1 already has no effect for autotools.

In fact I'm not sure about autotools. I suppose it depends on the
version of autotools that is used by the package.
Do you know an autotools package that currently takes Buildroot's V=1
into account?

And what about the other pkg-infra?
Does one of them take VERBOSE into account?

If we have too much doubts, we can also simply remove KBUILD_VERBOSE and
quiet, and keep on exporting VERBOSE when not empty. I think it would
just have been more clean to apply pkg-infra specific rules.

Let me know what you think about it...
Thank you.

Regards.

[Arnout Vandecappelle]

Hi Cédric,

Hi!
I'm getting back to working on this old subject (it was discussed in June).
I have proposed a first patch to fix the CMake specific problem.
http://git.buildroot.net/buildroot/commit/?id=307029867b0384446cd74b232b45a7b4f40cf0d1
VERBOSE is not exported anymore, unless V=1 in the command line.
It was suggested that I should provide a second patch, to address the problem
globally.
I have a local patch for that, but I need some feedback before providing it.
The idea is to remove exported variables that come from the kernel and are
useless in our context (KBUILD_VERBOSE and quiet), to remove VERBOSE, and to
manage the verbosity in a specific way in each pkg-infra (VERBOSE=1 is
understood by CMake, but might not be understood by another build system).

Yeah, it's not a good idea for us to export anything since we don't control our
child build systems. That export was most likely copied from the kernel's
Makefile years ago, without thinking about the implications.

When V=1, we should export Q = (empty) and a new variable BR2_VERBOSE = 1.

I don't think we should export anything at all, we should really do it per
build system (which is problematic for generic-package of course).

Q behaviour is not modified.
BR2_VERBOSE is taken into account in pkg-infra.
When BR2_VERBOSE=1, I add VERBOSE=1 in <PKG>_BUILD_CMDS
I don't know what to do. VERBOSE=1 has no effect, V=0/1 has no effect,
--enable-silent-rules has no effect.

For autotools it depends a lot on the version. Relatively recent automake (at
least 1.15) has the V=0/1 option, but it defaults to 1 I think. And of course,
the many packages that use autoconf but no automake do not support the V option.

NB: The only verbosity setting that has an effect on autotools is "make -s", but
it is different from "make V=0/1", and it is already taken into account by
Buildroot (QUIET variable in root Makefile).
* V=1 involves VERBOSE=1 for CMake.
* V=1 already has no effect for autotools.
In fact I'm not sure about autotools. I suppose it depends on the version of
autotools that is used by the package.
Do you know an autotools package that currently takes Buildroot's V=1 into account?

libpthsem for example.

And what about the other pkg-infra?
Does one of them take VERBOSE into account?

The kernel is an obvious one... And also derived ones, like busybox, uclibc,
uboot. But of course, it means it would have to be done for each individual
generic package...

So maybe exporting V is not such a bad idea after all. VERBOSE and Q I wouldn't
export, however.


Regards,
Arnout

If we have too much doubts, we can also simply remove KBUILD_VERBOSE and quiet,
and keep on exporting VERBOSE when not empty. I think it would just have been
more clean to apply pkg-infra specific rules.
Let me know what you think about it...
Thank you.
Regards.

[Cédric Marie]

Hi,

Post by Arnout Vandecappelle

When V=1, we should export Q = (empty) and a new variable BR2_VERBOSE = 1.

I don't think we should export anything at all, we should really do it per
build system (which is problematic for generic-package of course).

You're right for Q. It is used in pkg-generic.mk, and should be defined
only in this file, depending on V value and origin.

The benefits to exporting a new variable BR2_VERBOSE, is that we can
check V value (0 or 1) and origin ("command line"?) only once, instead
of repeating this double test in every .mk.

So now, thanks to your remarks, I've got something that is working for
cmake and autotools.

* I have removed these exports from root Makefile: quiet Q KBUILD_VERBOSE.

* In pkg-generic.mk, I define Q=(empty) if V=1 from the command line,
and Q=@ otherwise.

* In pkg-cmake.mk, I add "VERBOSE=1" in build command when V=1 from the
command line.
(checked with libcuefile)

* In pkg-autotools.mk, I add "V=1" in build command when V=1 from the
command line (and "V=0" otherwise, because V=1 is the default).
(checked with libpthsem)
NB: It would not even be necessary to modify pkg-autotools.mk, if V=0
was the default.

I think that these modifications keep the behaviour unchanged.

Can I provide a patch with that?

Do you prefer to check V value and origin in all pkg-*.mk, or should I
use BR2_VERBOSE to compute it once?

Waiting for your feedback,
Regards.

[Thomas Petazzoni]

Arnout, Cédric,

Post by Arnout Vandecappelle
For autotools it depends a lot on the version. Relatively recent automake (at
least 1.15) has the V=0/1 option, but it defaults to 1 I think.

It actually depends on how AM_SILENT_RULES is used:

* If it's not used, then the build is verbose, and V=0/1 has no effect.

* If AM_SILENT_RULES is used, with no special argument, then it
defaults to verbose (V=1), but can be made quiet by passing V=0

* If AM_SILENT_RULES([yes]) is used, then it defaults to quiet (V=0)
and can be made verbose by passing V=1.

Best regards,

Thomas

[Cédric Marie]

Hi,

Post by Thomas Petazzoni
Arnout, Cédric,

Post by Arnout Vandecappelle
For autotools it depends a lot on the version. Relatively recent automake (at
least 1.15) has the V=0/1 option, but it defaults to 1 I think.

* If it's not used, then the build is verbose, and V=0/1 has no effect.
* If AM_SILENT_RULES is used, with no special argument, then it
defaults to verbose (V=1), but can be made quiet by passing V=0
* If AM_SILENT_RULES([yes]) is used, then it defaults to quiet (V=0)
and can be made verbose by passing V=1.

So I think the best way to manage it is to be explicit: "V=0" when
nothing is specified (because we want the default to be silent), and
"V=1" when V=1 from the command line.
In worst case, it is not taken into account.
Otherwise, it is correctly managed, whatever the default setting of
AM_SILENT_RULE.

NB: I said that the behaviour was kept unchanged, but in fact there is
one case that seems to be better handled with my modification: when
AM_SILENT_RULES is used without argument (i.e. the default is verbose).
Now we turn it to quiet by forcing explicit V=0.

[Arnout Vandecappelle]

Hi,

Post by Thomas Petazzoni
Arnout, Cédric,

Post by Arnout Vandecappelle
For autotools it depends a lot on the version. Relatively recent automake (at
least 1.15) has the V=0/1 option, but it defaults to 1 I think.

* If it's not used, then the build is verbose, and V=0/1 has no effect.
* If AM_SILENT_RULES is used, with no special argument, then it
defaults to verbose (V=1), but can be made quiet by passing V=0
* If AM_SILENT_RULES([yes]) is used, then it defaults to quiet (V=0)
and can be made verbose by passing V=1.

So I think the best way to manage it is to be explicit: "V=0" when nothing is
specified (because we want the default to be silent), and "V=1" when V=1 from
the command line.

Actually, we want the default to be the package build system's default, I
think. So do nothing when nothing is specified, and add V=1 when V=1 is specified.

In worst case, it is not taken into account.
Otherwise, it is correctly managed, whatever the default setting of AM_SILENT_RULE.
NB: I said that the behaviour was kept unchanged, but in fact there is one case
that seems to be better handled with my modification: when AM_SILENT_RULES is
used without argument (i.e. the default is verbose). Now we turn it to quiet by
forcing explicit V=0.

Perhaps we could do that when V=0 is passed explicitly on buildroot's top-level
make.

BTW, this stuff should also be tested from an output directory and with a
non-022 umask.

Regards,
Arnout

[Cédric Marie]

Hi,

I've just sent the patch with my modifications.

Post by Arnout Vandecappelle

Post by Cédric Marie
we turn it to quiet by
forcing explicit V=0.

Perhaps we could do that when V=0 is passed explicitly on buildroot's top-level
make.

That's what I've done. I'm not sure whether it should be added in
Buildroot doc (8.1: make tips) or not, since it is specific to autotools
and has no effect on other packages.

Post by Arnout Vandecappelle
BTW, this stuff should also be tested from an output directory and with a
non-022 umask.

I have checked with O=... and with a different umask: the V option is
correctly forwarded.