Re: [hackers] [sbase][PATCH v2] Add tests for some utilities

From: Mattias Andrée <>
Date: Wed, 1 Aug 2018 19:53:18 +0200

Hi Silvan,

Thank you for your time!

The common code is 590 lines of code, including:

* 102 lines of code related to identifying the error when the
  test fails.

* 14 lines of code for properly killing processes on failure,
  abortion, and when a test case hangs.

* 32 lines of code, plus 13 of the lines counted above,
  for supporting concurrent tests.

This leaves 442 lines for the fundamental stuff and a few
lines to support printing all errors rather than just the
first error.

Some note on your sh code (since you wrote “crappy and
probably non-portable”, you are probably aware of this,
but just in case):

* `source` is a Bash synonym for the portable `.` command.

* `echo` is unportable and `printf` should be used instead.

* I have never seen `&>` before, so I my guess is that it
  is a Bashism.

* It looks like whichever file descriptor is not being
  inspected by `check_output` is printed the inherited
  file descriptor rather than to /dev/null.

* I think it would be better to have something like:

        set -e

        run "test name" "./dirname //"
        check_stderr ""
        check_stdout / || check_stdout //
        check_exit 0

Your sh code, with check_exit added, covers most current tests.
However, it every time the usage output is modified it has to
be change in the test case, which I guess is acceptable but
undesirable. The tests that are currently implemented
and which need to be handled specially are:

* sleep:
        This can be done with sh. With some adaption to the sh
        code, tests can also be done in parallel as it is in
        the C code.

* test:
        test takes a lot of time to test, which is why multiple
        tests are run in parallel in the C code. Like tty(1),
        this test also requires the creation of terminals, but
        it also requires the creation of sockets.

* time:
        Requires setitimer(3) and pause(3).

* tty:
        This test requires the creation of terminals.

* uname:
        Most of uname can be tested in ed, however, the output
        of uname with only one flag applied requires the uname
        system call.

* whoami:
        The user name can be retrieved via $LOGNAME according
        to POSIX, however this requires that your login program
        actually sets it. Additionally (and this should be added
        to the test) when whoami is called from a program with
        setuid the owner of the program should be printed (i.e.
        the effective user), not real user which is stored in

Additionally env, time, and yes requires identifying which signal
the processed was killed by. I have never done this in sh, but I
understand that it should be doable. time and env also requires a
way to kill the process it runs with a specific signal. Furthermore
the test for tty should include a case where stdin does not exist,
which for the moment is not done. All tests excepts test and sleep
fundamentally requires support for stdout, but they also use stderr
for checking error support.

These tests require all the features in the C code, except the
extra functionally enumerated at the beginning of this mail and
the support for binary data. These are tests we should focus on,
whichever solution is found for them should be applied to the
rest tests since those require almost only subset of the
functionality. The only extra functionality other tests require
is support for binary data; which I really don't think warrant
a separate solution.

The following utilities (from both sbase and ubase) still need
tests, as well as the utilities listed under ubase's TODO. I
have most of the work already done for the utilities marked
with an asterisk.

        at awk bc cal
        cat chgrp chmod chown
        chroot chvt cksum* clear
        cmp cols comm cp
        cron ctrlaltdel cut date
        dd df diff dmesg
        du ed eject env
        expr fallocate find flock
        fold free freeramdisk fsfreeze
        getconf getty grep halt
        head hostname hwclock id
        insmod install join kill
        killall5 last lastlog ln
        logger login logname ls
        lsmod lsusb md5sum* mesg
        mkdir mkfifo mknod mkswap
        mktemp mount mountpoint mv
        nice nl nohup nologin*
        od pagesize passwd paste
        patch* pathchk pidof pivot_root
        printf ps pwd pwdx
        readahead readlink renice respawn
        rev rm rmdir rmmod
        sed seq* setsid sha1sum*
        sha224sum* sha256sum* sha384sum* sha512-224sum*
        sha512-256sum* sha512sum* sort split
        sponge stat strings stty
        su swaplabel swapoff swapon
        switch_root sync sysctl tail
        tar tee tftp touch
        tr truncate tsort umount
        uniq* unshare uptime uudecode
        uuencode vtallow watch wc
        which who xargs

I think should look into what their tests will require, maybe
even write some tests with optional test framework, especially
for the tests that cannot be trivially tested in sh. We should
also identify which of them cannot be automatically tested
without a virtual machine, and write them down in the README,
and possibly list of features to test. Once this is done, we
can make a more informed discussion.

For the moment I see two options: 1) write the tests mostly
in sh and write special utilities in C where required, and
2) use the C code, but look for ways to improve it.

As I see it, the most complex parts of the C code are:

* start_process:
        It's probably enough to split out some code to
        separate functions: the `if (in->flags & DATA)`,
        the dup2–close loops at the end.

* wait_process:
        Instead of ready all file descriptors as fast as
        possible, the they could probably be read in order.

* check_output_test:
        It's probably enough to add a few short comments
        and improve variable names.

* print_failure:
        It's probably enough to add some empty lines add a
        few short comments.

The other parts are pretty straight forward.

Mattias Andrée

On Wed, 1 Aug 2018 16:36:35 +0200
Silvan Jegen <> wrote:

> Hi Mattias!
> On Wed, Jul 11, 2018 at 09:39:23PM +0200, Mattias Andrée wrote:
> > The following utilities are tested:
> > - basename(1)
> > - dirname(1)
> > - echo(1)
> > - false(1)
> > - link(1)
> > - printenv(1)
> > - sleep(1)
> > - test(1)
> > - time(1)
> > - true(1)
> > - tty(1)
> > - uname(1)
> > - unexpand(1)
> > - unlink(1)
> > - whoami(1)
> > - yes(1)
> >
> > Some tests contain "#ifdef TODO", these tests current
> > fail, but there are patches submitted for most of them.
> > There are not patches submitted for fixing the
> > "#ifdef TODO"s in expand.test.c and unexpand.test.c.
> >
> > Signed-off-by: Mattias Andrée <>
> Sorry for not getting around to looking at this earlier.
> I definitely think we should have unit tests for sbase (and other
> projects?) as soon as possible. What concerns me with your approach is
> that we have about 700 lines of C code in testing-common.{c,h} of which
> I feel quite a bit could be dropped.
> I have written some (crappy and probably non-portable) shell script
> functions to check the stdout and stderr of a process. It's about 40
> lines. I also converted your tests for dirname to use these functions
> (both files attached. The test coverage is not exactly the same but
> relatively similar).
> I wonder if we couldn't use some cleaned-up version of the shell script
> functions for the easy test cases that only check stdout and stderr
> output and your custom C code for the more specialised test cases (like
> 'tty').
> What do you think?
> Cheers,
> Silvan
> > ---
> > Makefile | 45 ++++-
> > basename.test.c | 68 +++++++
> > dirname.test.c | 55 ++++++
> > echo.test.c | 51 ++++++
> > expand.test.c | 92 ++++++++++
> > false.test.c | 32 ++++
> > link.test.c | 58 ++++++
> > printenv.test.c | 79 ++++++++
> > sleep.test.c | 53 ++++++
> > test-common.c | 560 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > test-common.h | 219 ++++++++++++++++++++++
> > test.test.c | 408 +++++++++++++++++++++++++++++++++++++++++
> > time.test.c | 218 ++++++++++++++++++++++
> > true.test.c | 31 ++++
> > tty.test.c | 44 +++++
> > uname.test.c | 283 ++++++++++++++++++++++++++++
> > unexpand.test.c | 97 ++++++++++
> > unlink.test.c | 56 ++++++
> > whoami.test.c | 38 ++++
> > yes.test.c | 131 +++++++++++++
> > 20 files changed, 2614 insertions(+), 4 deletions(-)
> >
> > [snip]

Received on Wed Aug 01 2018 - 19:53:18 CEST

This archive was generated by hypermail 2.3.0 : Wed Aug 01 2018 - 20:00:22 CEST