On Sun, Jul 08, 2018 at 11:13:52AM +0200, Mattias Andrée wrote:
> On Sun, 8 Jul 2018 10:06:14 +0100
> Dimitris Papastamos <sin_AT_2f30.org> wrote:
>
> > On Sun, Jul 08, 2018 at 01:12:59AM +0200, Mattias Andrée wrote:
> > > On Sat, 7 Jul 2018 23:29:53 +0100
> > > Dimitris Papastamos <sin_AT_2f30.org> wrote:
> > >
> > > > On Sun, Jul 08, 2018 at 12:12:08AM +0200, Mattias Andrée wrote:
> > > > > On Sat, 7 Jul 2018 22:55:28 +0100
> > > > > Dimitris Papastamos <sin_AT_2f30.org> wrote:
> > > > >
> > > > > > This is too intrusive, what's wrong with using a shell script to test
> > > > > > the commands?
> > > > > >
> > > > > > The test framework is more complicated than most sbase commands.
> > > > > >
> > > > > > It would have been nice to discuss this in advance before writing a
> > > > > > 1000 line patch that might not get merged.
> > > > > >
> > > > >
> > > > > Writing all tests in shell isn't the best idea I think.
> > > > > This frameworks makes it easy to write test and it will
> > > > > tell you everything you need to know to figure out what
> > > > > failed. I believe that in the need this will reduce the
> > > > > amount of test code. There are things that are difficult
> > > > > to do in shell: for example create a terminal which is
> > > > > need to test tty(1). Look for example at the tests in
> > > > > https://github.com/maandree/base-util-tests, they aren't
> > > > > that nice, and they even require bash(1), it would be
> > > > > even worse with portable sh(1).
> > > >
> > > > so what you are saying is that your shell code sucks so
> > > > it is better done in C?
> > > >
> > > > sorry im not buying it, this is overkill
> > > >
> > >
> > > No, I'm saying that if you want to do it in sh(1) you will
> > > need more test, or make really crappy helper functions, and
> > > you will also need to write special utilities in C just to
> > > test some utilities that cannot be tested entirely in sh(1).
> > >
> > > The C code contains:
> > >
> > > * a way to print where in a loop a test fails.
> > > this will be useful for example when when adding test
> > > cases for the *sum utilities,
> > >
> > > * a way to make tests asynchronously, this could be removed
> > > but will be useful for testing sleep(1), so it does not
> > > take too long,
> > >
> > > * a simple way to measure and test how long a test took,
> > >
> > > * a set of tiny functions to declare how to program shall
> > > be started,
> > >
> > > * a way to create sockets and TTYs. The socket part can
> > > probably be removed but it would be useful for testing
> > > different file types. Support for regular files should
> > > probably be added. TTYs are required to testing tty(1),
> > >
> > > * ways to test the exit status with support for both normal
> > > exit and kill by signal,
> > >
> > > * ways to check the output to stdout and stderr. In sh(1),
> > > test(1) and grep(1) could be used,
> > >
> > > * some code to make to test cases smaller,
> > >
> > > * a function to spawn a process with the requested files
> > > and input, and
> > >
> > > * a function to read a process's output and wait for it.
> > >
> > > I wouldn't say that it's complex, its just a few functions, and
> > > 2 or 3 of them is are bit long, but not particularly long. I
> > > think this is worth it for the consistence, short tests and
> > > easily readable tests, a convenient way to locate the failing
> > > test and what's actually wrong, and to not have extra utilities
> > > (also written in C) to do the parts of the tests that cannot be
> > > done in C. I'm not sure how mean that the code is complicated,
> > > it's just long.
> >
> > https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/
> >
> > whatever you do, this test code should be in a different repo
> > like sbase-regress or similar.
> >
>
> I will reduce code in the test-common.[ch].
>
> If the tests are in a separate repository, package maintainers
> need to download both, do you really think this is a good idea?
> What's the advantage with make it a separate repository?
the tests are primarily for the developers, no user cares if you can
pass all the internal regression tests but fail to build firefox.
time is better spent writing useful test cases rather than some "test
framework".
Received on Sun Jul 08 2018 - 11:19:46 CEST