Re: [dev] ii: how to process out in a pipeline and still page with less

From: Kyryl Melekhin <k.melekhin_AT_gmail.com>
Date: Fri, 27 May 2022 16:23:04 +0000

"Greg Reagle" <list_AT_speedpost.net> wrote:

> I have a file named "out" (from ii) that I want to view. Of course, it can
> grow while I am viewing it. I can view it with "tail -f out" or "less +F out
> ", both of which work. I also want to apply some processing in a pipeline,
> something like "tail -f out | tr a A | less" but that does not work. The
> less command ignores my keystrokes (unless I hit Ctrl-C, but that kills tail
> and tr). The "tr a A" command is arbitrary; you can substitute whatever
> processing you want, even just cat.

Hello Greg, this is a fundamental limitation of stdin and the tty
subsystem. A while ago I had my head banging against a wall trying to
write an application that can read keyboard input from the stdin
after all the data has been drained (read) from the pipe. Turns out
it is impossible to reset the descriptor fd that is used for reading
the stdin to the state as if there was no pipe at all. I had to resort
to looking over implementation of unix program less to find out
how they do the keyboard input such that it can handle both inputs
at the same time. What they do is just swap between different special
fd's one for keyboard and other one for stdin. However even after
taking apart the code from less into a smaller test program I never
been able to reproduce the same behavior that of less. If using
the poll() function to wait for keyboard input, if the program is
piped into the poll(); call becomes nonblocking even if special
descriptors are setup using dup2(); calls or changing some flags
using fcntl();. Till this day I have no idea how to implement
this specific behavior of less and if I be honest it's fucking
black magic. So I suspect the reason you can't use less in that
scenario you described is exactly because of this black magic
that onbody knows about since there is absolutely no documentation
on how to make it work.

Anybody else reading this, if you be kind enough to provide a
small sample C program that can read the data from the pipe, ie
do echo "hello world" | ./a.out and be able to recover after reading
the "hello world", put the stdin back into "blocking" mode so that
it starts to wait for keyboard input after that just like it would do
if there was no pipe thing going on. Also as a bonus, your stdout
should remain unaffected, so you should be able to just write(1, "hello world\n", ...);
being displayed as expected.
And this must be done using only functions specified in posix/linux kernel,
so no fgets, no fgetc or any other input reading function that hides what is
actually going on. Use functions like open(), fcntl(), poll(), dup() only.

I believe once this mystery is uncovered it would be possible to
solve your issue with less, since it would be clear on how
to manipulate the stdin descriptor to do virtually anything we want.

I also remember reading some neckbeard guy post about how less
is not technically a unix program because of how it interfaces
with the pipes and stdin, whatever that means. Maybe we are
all trying to do things that were not originally envisioned
on this interface. It is either you read from pipe or you
read from keyboard, having both at the same time - heresy.

> This command "tail -f out | tr a A" is functional and has no bugs, but it
> doesn't let me use the power of less, which I crave.
>
> This command "tail out | tr a A | less" is functional and has no bugs, but
> it doesn't let me see newly appended lines.
>
> Can I use the power of the Unix pipeline to do text processing and the
> power of less for excellent paging and still be able to see new lines as they
> are appended? Why doesn't or can't less continue to monitor stdin from the
> pipeline and respond to my keystrokes from the tty?
>
> I am using Debian 11 in case it matters, with fish. But I am happy to try
> other shells. In fact I already have and that doesn't seem to help. I have
> also tried more, most, nano -, and vi -, instead of less, to no avail.

Unsuprising.

Best wishes,
Kyryl
Received on Fri May 27 2022 - 18:23:04 CEST