[Buildroot] Issue with toolchain wrapper changes

[Thomas Petazzoni]

Hello Arnout,

I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.

As you know, Buildroot installs all its host stuff in $(HOST_DIR)/usr,
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
to look like conventional toolchains, at the end of their build, I do:

mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/

Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being
executed. Example:

***@skate:~/x-tools/armv5-musl-new-2016.02-1$ ls
arm-buildroot-linux-musleabi bin include lib libexec share
***@skate:~/x-tools/armv5-musl-new-2016.02-1$ ./bin/arm-linux-gcc -v
/home/thomas/x-tools/usr/bin/arm-linux-gcc.br_real: No such file or directory

As you can see, it looks for arm-linux-gcc.br_real
in /home/thomas/x-tools/usr/bin, while it is
in /home/thomas/x-tools/armv5-musl-new-2016.02-1/bin/.

If I move everything again under a usr/ subdirectory, it all works fine
again:

***@skate:~/x-tools/armv5-musl-new-2016.02-1$ mkdir usr
***@skate:~/x-tools/armv5-musl-new-2016.02-1$ mv * usr/
mv: cannot move ‘usr’ to a subdirectory of itself, ‘usr/usr’
***@skate:~/x-tools/armv5-musl-new-2016.02-1$ ./usr/bin/arm-linux-gcc -v
Using built-in specs.
COLLECT_GCC=/home/thomas/x-tools/armv5-musl-new-2016.02-1/usr/bin/arm-linux-gcc.br_real
COLLECT_LTO_WRAPPER=/home/thomas/x-tools/armv5-musl-new-2016.02-1/usr/bin/../libexec/gcc/arm-buildroot-linux-musleabi/5.3.0/lto-wrapper
Target: arm-buildroot-linux-musleabi
Configured with: ./configure --prefix=/build/armv5-musl-new-2016.02-1/usr --sysconfdir=/build/armv5-musl-new-2016.02-1/etc --enable-static --target=arm-buildroot-linux-musleabi --with-sysroot=/build/armv5-musl-new-2016.02-1/usr/arm-buildroot-linux-musleabi/sysroot --disable-__cxa_atexit --with-gnu-ld --disable-libssp --disable-multilib --with-gmp=/build/armv5-musl-new-2016.02-1/usr --with-mpfr=/build/armv5-musl-new-2016.02-1/usr --with-pkgversion='Buildroot 2016.02-rc1-00006-gbb13242' --with-bugurl=http://bugs.buildroot.net/ --disable-libquadmath --disable-libsanitizer --disable-tls --disable-libmudflap --enable-threads --with-mpc=/build/armv5-musl-new-2016.02-1/usr --without-isl --without-cloog --with-float=soft --disable-decimal-float --with-abi=aapcs-linux --with-cpu=arm926ej-s --with-float=soft --with-mode=arm --enable-languages=c,c++ --with-build-time-tools=/build/armv5-musl-new-2016.02-1/usr/arm-buildroot-linux-musleabi/bin --enable-shared --disable-libgomp
Thread model: posix
gcc version 5.3.0 (Buildroot 2016.02-rc1-00006-gbb13242)

Now, I'm sure the problem is at the beginning of the
toolchain-wrapper.c main() function. But I'm wondering what is the
proper fix for this. Should I fix the toolchain-wrapper logic to
properly work in such a situation ? Should we remove the usr/
sub-directory completely from Buildroot, as you suggested multiple
times already ?

Do you have some other suggestions ?

Thanks!

Thomas

[Arnout Vandecappelle]

Post by Thomas Petazzoni
Hello Arnout,
I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.
As you know, Buildroot installs all its host stuff in $(HOST_DIR)/usr,
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/
Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being
arm-buildroot-linux-musleabi bin include lib libexec share
/home/thomas/x-tools/usr/bin/arm-linux-gcc.br_real: No such file or directory
As you can see, it looks for arm-linux-gcc.br_real
in /home/thomas/x-tools/usr/bin, while it is
in /home/thomas/x-tools/armv5-musl-new-2016.02-1/bin/.

5ce73dca5238d30b0cbfe64c1ecbaec777c6a8fe was supposed to fix this situation. I
didn't find a way to avoid hardcoding the BR_CROSS_PATH_SUFFIX/usr/bin bit in
the wrapper, so instead I reversed the logic: the _external_ toolchain wrapper
will avoid calling the wrapper and instead call the .br_real directly. So,
calling the wrapper outside the usr directory will indeed fail, but when it's
used as an external toolchain in buildroot, it should work fine.

Obviously this is not an ideal situation, but I don't have better ideas. Well,
except to get rid of the usr part, as you point out below. The problem is that
the wrapper needs to have absbasedir to find the sysroot, and for the external
toolchain this needs to lead to opt/ext-toolchain. Well, maybe we coud use ../
in BR_SYSROOT, or we could make more than a difference between the internal and
the external toolchain, or we could add keep a bindir variable in addition to
absbasedir and use that for ccache and the internal toolchain. Right, so I do
have some ideas :-)

Anyway it's going to be a much bigger change than the one from 5ce73dca, and I
think it's not for 2016.02.


Regards,
Arnout

Post by Thomas Petazzoni
If I move everything again under a usr/ subdirectory, it all works fine
mv: cannot move ‘usr’ to a subdirectory of itself, ‘usr/usr’
Using built-in specs.
COLLECT_GCC=/home/thomas/x-tools/armv5-musl-new-2016.02-1/usr/bin/arm-linux-gcc.br_real
COLLECT_LTO_WRAPPER=/home/thomas/x-tools/armv5-musl-new-2016.02-1/usr/bin/../libexec/gcc/arm-buildroot-linux-musleabi/5.3.0/lto-wrapper
Target: arm-buildroot-linux-musleabi
Configured with: ./configure --prefix=/build/armv5-musl-new-2016.02-1/usr --sysconfdir=/build/armv5-musl-new-2016.02-1/etc --enable-static --target=arm-buildroot-linux-musleabi --with-sysroot=/build/armv5-musl-new-2016.02-1/usr/arm-buildroot-linux-musleabi/sysroot --disable-__cxa_atexit --with-gnu-ld --disable-libssp --disable-multilib --with-gmp=/build/armv5-musl-new-2016.02-1/usr --with-mpfr=/build/armv5-musl-new-2016.02-1/usr --with-pkgversion='Buildroot 2016.02-rc1-00006-gbb13242' --with-bugurl=http://bugs.buildroot.net/ --disable-libquadmath --disable-libsanitizer --disable-tls --disable-libmudflap --enable-threads --with-mpc=/build/armv5-musl-new-2016.02-1/usr --without-isl --without-cloog --with-float=soft --disable-decimal-float --with-abi=aapcs-linux --with-cpu=arm926ej-s --with-float=soft --with-mode=arm --enable-languages=c,c++ --with-build-time-tools=/build/armv5-musl-new-2016.02-1/usr/arm-buildroot-linux-musleabi/bin --enable-shared --disable-libgomp
Thread model: posix
gcc version 5.3.0 (Buildroot 2016.02-rc1-00006-gbb13242)
Now, I'm sure the problem is at the beginning of the
toolchain-wrapper.c main() function. But I'm wondering what is the
proper fix for this. Should I fix the toolchain-wrapper logic to
properly work in such a situation ? Should we remove the usr/
sub-directory completely from Buildroot, as you suggested multiple
times already ?
Do you have some other suggestions ?
Thanks!
Thomas

[Thomas Petazzoni]

Arnout,

Post by Arnout Vandecappelle
5ce73dca5238d30b0cbfe64c1ecbaec777c6a8fe was supposed to fix this situation. I
didn't find a way to avoid hardcoding the BR_CROSS_PATH_SUFFIX/usr/bin bit in
the wrapper, so instead I reversed the logic: the _external_ toolchain wrapper
will avoid calling the wrapper and instead call the .br_real directly. So,
calling the wrapper outside the usr directory will indeed fail, but when it's
used as an external toolchain in buildroot, it should work fine.
Obviously this is not an ideal situation, but I don't have better ideas. Well,
except to get rid of the usr part, as you point out below. The problem is that
the wrapper needs to have absbasedir to find the sysroot, and for the external
toolchain this needs to lead to opt/ext-toolchain. Well, maybe we coud use ../
in BR_SYSROOT, or we could make more than a difference between the internal and
the external toolchain, or we could add keep a bindir variable in addition to
absbasedir and use that for ccache and the internal toolchain. Right, so I do
have some ideas :-)

I started looking a bit at the issue. My idea is to change the wrapper
to work with absbindir instead of absbasedir. absbindir is the absolute
path to the binary tool themselves.

It generally works quite fine, except for the sysroot, where it becomes
tricky. The BR_SYSROOT variable was passed as usr/<tuple>/sysroot,
which no longer works when usr/ is removed. So I've changed that to a
BR_REL_SYSROOT definition, which must be passed when building the
toolchain wrapper as the relative path between the binary tool
directory and the staging directory.

The other tricky case is the external toolchain case with relative path
(i.e the external toolchain was extracted in host/opt/ext-toolchain/).
I have to add a hardcoded ../../ in this case, but I believe this is OK
because the usr/ directory can anyway not be removed from the host
directory when an external toolchain is used, since opt/ext-toolchain/
is outside of the usr/ directory. Alternatively, we could decide to
extract the external toolchain somewhere in usr/ rather than in opt/,
so that really everything in $(HOST_DIR) is inside this usr/ directory.

It gives the following logic (I'm interested by your
comments/suggestions) :

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index 887058f..43ae648 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -22,6 +22,7 @@
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
+#include <libgen.h>

#ifdef BR_CCACHE
static char ccache_path[PATH_MAX];
@@ -102,7 +103,7 @@ static void check_unsafe_path(const char *path, int paranoid)
int main(int argc, char **argv)
{
char **args, **cur, **exec_args;
- char *relbasedir, *absbasedir;
+ char *relbasedir, *absbindir;
char *progpath = argv[0];
char *basename;
char *env_debug;
@@ -115,55 +116,44 @@ int main(int argc, char **argv)
if (basename) {
*basename = '\0';
basename++;
- relbasedir = malloc(strlen(progpath) + 7);
- if (relbasedir == NULL) {
- perror(__FILE__ ": malloc");
- return 2;
- }
- sprintf(relbasedir, "%s/../..", argv[0]);
- absbasedir = realpath(relbasedir, NULL);
+ absbindir = realpath(argv[0], NULL);
} else {
+ char *tmp;
basename = progpath;
- absbasedir = malloc(PATH_MAX + 1);
- ret = readlink("/proc/self/exe", absbasedir, PATH_MAX);
+ absbindir = malloc(PATH_MAX + 1);
+ ret = readlink("/proc/self/exe", absbindir, PATH_MAX);
if (ret < 0) {
perror(__FILE__ ": readlink");
return 2;
}
- absbasedir[ret] = '\0';
- for (i = ret; i > 0; i--) {
- if (absbasedir[i] == '/') {
- absbasedir[i] = '\0';
- if (++count == 3)
- break;
- }
- }
+ absbindir[ret] = '\0';
+ absbindir = dirname(absbindir);
}
- if (absbasedir == NULL) {
+ if (absbindir == NULL) {
perror(__FILE__ ": realpath");
return 2;
}

/* Fill in the relative paths */
#ifdef BR_CROSS_PATH_REL
- ret = snprintf(path, sizeof(path), "%s/" BR_CROSS_PATH_REL "/%s" BR_CROSS_PATH_SUFFIX, absbasedir, basename);
+ ret = snprintf(path, sizeof(path), "%s/../../" BR_CROSS_PATH_REL "/%s" BR_CROSS_PATH_SUFFIX, absbindir, basename);
#elif defined(BR_CROSS_PATH_ABS)
ret = snprintf(path, sizeof(path), BR_CROSS_PATH_ABS "/%s" BR_CROSS_PATH_SUFFIX, basename);
#else
- ret = snprintf(path, sizeof(path), "%s/usr/bin/%s" BR_CROSS_PATH_SUFFIX, absbasedir, basename);
+ ret = snprintf(path, sizeof(path), "%s/%s" BR_CROSS_PATH_SUFFIX, absbindir, basename);
#endif
if (ret >= sizeof(path)) {
perror(__FILE__ ": overflow");
return 3;
}
#ifdef BR_CCACHE
- ret = snprintf(ccache_path, sizeof(ccache_path), "%s/usr/bin/ccache", absbasedir);
+ ret = snprintf(ccache_path, sizeof(ccache_path), "%s/ccache", absbindir);
if (ret >= sizeof(ccache_path)) {
perror(__FILE__ ": overflow");
return 3;
}
#endif
- ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_SYSROOT, absbasedir);
+ ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_REL_SYSROOT, absbindir);
if (ret >= sizeof(sysroot)) {
perror(__FILE__ ": overflow");
return 3;
diff --git a/toolchain/toolchain-wrapper.mk b/toolchain/toolchain-wrapper.mk
index af39071..8725397 100644
--- a/toolchain/toolchain-wrapper.mk
+++ b/toolchain/toolchain-wrapper.mk
@@ -10,7 +10,7 @@ TOOLCHAIN_WRAPPER_HASH_STYLE = both
endif

TOOLCHAIN_WRAPPER_ARGS = $($(PKG)_TOOLCHAIN_WRAPPER_ARGS)
-TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"'
+TOOLCHAIN_WRAPPER_ARGS += -DBR_REL_SYSROOT='"../$(GNU_TARGET_NAME)/sysroot"'

# We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a
# separate argument when used in execv() by the toolchain wrapper.



Thomas

[Peter Korsgaard]

Post by Thomas Petazzoni
Hello Arnout,
I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.
As you know, Buildroot installs all its host stuff in $(HOST_DIR)/usr,
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/
Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being

Is the extra /usr really such a problem? I've built a bunch of
toolchains, and just refer to them as /opt/br/<whatever>/usr.

[Thomas Petazzoni]

Hello,

Post by Peter Korsgaard

Post by Thomas Petazzoni
Hello Arnout,
I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.
As you know, Buildroot installs all its host stuff in $(HOST_DIR)/usr,
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/
Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being

Is the extra /usr really such a problem? I've built a bunch of
toolchains, and just refer to them as /opt/br/<whatever>/usr.

Yes, for my purpose it's annoying. I want to produce pre-built
toolchains that everyone can use, in Buildroot or something else, so I
want them to look like normal toolchains. And normal toolchains don't
have this useless usr/ top-level directory.

Best regards,

Thomas

[Thomas De Schampheleire]

On Feb 17, 2016 13:00, "Thomas Petazzoni" <

Post by Thomas Petazzoni
Hello,

Post by Peter Korsgaard

Post by Thomas Petazzoni
Hello Arnout,
I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.
As you know, Buildroot installs all its host stuff in

$(HOST_DIR)/usr,

Post by Thomas Petazzoni

Post by Peter Korsgaard

Post by Thomas Petazzoni
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/
Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being

Is the extra /usr really such a problem? I've built a bunch of
toolchains, and just refer to them as /opt/br/<whatever>/usr.

Yes, for my purpose it's annoying. I want to produce pre-built
toolchains that everyone can use, in Buildroot or something else, so I
want them to look like normal toolchains. And normal toolchains don't
have this useless usr/ top-level directory.

I may be misunderstanding something, but what is the problem with creating
the archive from the usr directory? This is what I did at the time I was
using a buildroot toolchain.

/Thomas

[Thomas Petazzoni]

Hello,

Post by Thomas De Schampheleire
I may be misunderstanding something, but what is the problem with creating
the archive from the usr directory? This is what I did at the time I was
using a buildroot toolchain.

It no longer works, see the initial e-mail I sent to start this thread.

Best regards,

Thomas

[Arnout Vandecappelle]

Post by Thomas Petazzoni
Hello,

Post by Peter Korsgaard

Post by Thomas Petazzoni
Hello Arnout,
I am facing some issues with the toolchain wrapper, which I believe
were introduced by the changes done to use the toolchain wrapper also
for the internal toolchain backend.
As you know, Buildroot installs all its host stuff in $(HOST_DIR)/usr,
and not directly under $(HOST_DIR). For the autobuilders, I build a
number of toolchains with Buildroot, which are then used as pre-built
external toolchains by the autobuilders. In order for those toolchains
mv /path/to/host/usr/* /path/to/host/
rmdir /path/to/host/usr/
Which moves everything outside of the usr/ subdirectory, and removes
the usr/ subdirectory itself. This used to work perfectly fine, but
now, it fails to find the .br_real equivalent of the command being

Is the extra /usr really such a problem? I've built a bunch of
toolchains, and just refer to them as /opt/br/<whatever>/usr.

Yes, for my purpose it's annoying. I want to produce pre-built
toolchains that everyone can use, in Buildroot or something else, so I
want them to look like normal toolchains. And normal toolchains don't
have this useless usr/ top-level directory.

I may be misunderstanding something, but what is the problem with creating the
archive from the usr directory? This is what I did at the time I was using a
buildroot toolchain.

It works fine when you use it as an external toolchain in buildroot, but it
doesn't work when you try to call cross-gcc directly. The wrapper will look for
the real executable in ../../usr/bin.

Regards,
Arnout