On Sun, Jul 08, 2018 at 01:12:59AM +0200, Mattias Andrée wrote:
> On Sat, 7 Jul 2018 23:29:53 +0100
> Dimitris Papastamos <sin_AT_2f30.org> wrote:
>
> > On Sun, Jul 08, 2018 at 12:12:08AM +0200, Mattias Andrée wrote:
> > > On Sat, 7 Jul 2018 22:55:28 +0100
> > > Dimitris Papastamos <sin_AT_2f30.org> wrote:
> > >
> > > > This is too intrusive, what's wrong with using a shell script to test
> > > > the commands?
> > > >
> > > > The test framework is more complicated than most sbase commands.
> > > >
> > > > It would have been nice to discuss this in advance before writing a
> > > > 1000 line patch that might not get merged.
> > > >
> > >
> > > Writing all tests in shell isn't the best idea I think.
> > > This frameworks makes it easy to write test and it will
> > > tell you everything you need to know to figure out what
> > > failed. I believe that in the need this will reduce the
> > > amount of test code. There are things that are difficult
> > > to do in shell: for example create a terminal which is
> > > need to test tty(1). Look for example at the tests in
> > > https://github.com/maandree/base-util-tests, they aren't
> > > that nice, and they even require bash(1), it would be
> > > even worse with portable sh(1).
> >
> > so what you are saying is that your shell code sucks so
> > it is better done in C?
> >
> > sorry im not buying it, this is overkill
> >
>
> No, I'm saying that if you want to do it in sh(1) you will
> need more test, or make really crappy helper functions, and
> you will also need to write special utilities in C just to
> test some utilities that cannot be tested entirely in sh(1).
>
> The C code contains:
>
> * a way to print where in a loop a test fails.
> this will be useful for example when when adding test
> cases for the *sum utilities,
>
> * a way to make tests asynchronously, this could be removed
> but will be useful for testing sleep(1), so it does not
> take too long,
>
> * a simple way to measure and test how long a test took,
>
> * a set of tiny functions to declare how to program shall
> be started,
>
> * a way to create sockets and TTYs. The socket part can
> probably be removed but it would be useful for testing
> different file types. Support for regular files should
> probably be added. TTYs are required to testing tty(1),
>
> * ways to test the exit status with support for both normal
> exit and kill by signal,
>
> * ways to check the output to stdout and stderr. In sh(1),
> test(1) and grep(1) could be used,
>
> * some code to make to test cases smaller,
>
> * a function to spawn a process with the requested files
> and input, and
>
> * a function to read a process's output and wait for it.
>
> I wouldn't say that it's complex, its just a few functions, and
> 2 or 3 of them is are bit long, but not particularly long. I
> think this is worth it for the consistence, short tests and
> easily readable tests, a convenient way to locate the failing
> test and what's actually wrong, and to not have extra utilities
> (also written in C) to do the parts of the tests that cannot be
> done in C. I'm not sure how mean that the code is complicated,
> it's just long.
https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/
whatever you do, this test code should be in a different repo
like sbase-regress or similar.
Received on Sun Jul 08 2018 - 11:06:14 CEST