RE: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

From: Mattias Andrée <maandree_AT_kth.se>
Date: Tue, 12 Dec 2017 11:02:11 +0000

Perhaps I should clarify that (1) the goal would be to have blind-tee
(and blind-cat if that is implemented) to use already existing functions
that sends data between two files, and have these functions use splice
when possible), so would only use tee explicitly, not splice, and (2)
a 1 hour long blind video with the resolution 1920x1080_AT_30 is 6.5TB
large, so we are talking about a massive amount of data that is beeing
sent between processes. In total, a 1 hour long video could require
houndreds of terabytes being sent between process. Double that if it is a
60 fps video (which you will actually on the Internet nowadays) and
multiple that by 4 for a 4K video, and double that again for a stereoscopic
video (you can find all of these things on Blu-ray videos). So we could
be talking about a couple of petabytes of data for a profession video,
and 100TB for an amateur video, not just a couple of gigabytes. Try to
copy that about of data with dd (in parallell of course), and you might
find that even halving that time[1] would be nice, even if most if the time
in the rendering process is usually spent on rendering effects, stacking
videos, and transcoding.

[1] 42 hours per petabyte.

________________________________________
From: Mattias Andrée [maandree_AT_kth.se]
Sent: 12 December 2017 11:32
To: hackers mail list
Subject: RE: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

When I rendered a video, tee used 100% while the other process was
basically at 0, for more than 50% of the rendering time. I ran it multiply
times to verify that is was correct. An alternative solution would be
to use sockets, but that would require changes to the shell. Optimising
tee seems like the sensible alternative. Besides, normally when you use
tools like tee and cat, you expect it to finish within milliseconds, even
for larger files, here we are talking about time intervals between seconds
and a few hours, with most if not all CPU:s at 90% to 100%, so optimisations
do not hurt, especially not when as significant as using tee and splice. It
may be unfortunate to have to use both tee–splice and read–write (which
is required both for platforms not supporting tee and splice, and for file
types not supporting them), but considering how little complexity this adds,
— it's not much more than a duplication of a function, — I think it is a
trade-off worth considering.

________________________________________
From: isabella parakiss [izaberina_AT_gmail.com]
Sent: 12 December 2017 10:58
To: hackers mail list
Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

bullshit

On 12/4/17, git_AT_suckless.org <git_AT_suckless.org> wrote:
> commit d8aa45da86d1128149fd7ab6ac3725bf8e88a1b1
> Author: Mattias Andrée <maandree_AT_kth.se>
> AuthorDate: Mon Dec 4 22:35:59 2017 +0100
> Commit: Mattias Andrée <maandree_AT_kth.se>
> CommitDate: Mon Dec 4 22:35:59 2017 +0100
>
> update todo: tee is too slow
>
> Signed-off-by: Mattias Andrée <maandree_AT_kth.se>
>
> diff --git a/TODO b/TODO
> index b0bbde7..408e942 100644
> --- a/TODO
> +++ b/TODO
> _AT_@ -1,3 +1,7 @@
> +blind-tee (and tee(1)) is too slow (bottleneck) and must be reimplemented
> +using tee(2) and splice(2). cat(1) may also be too slow, if this is the
> +case, add blind-splice that just copies stdin to stdout using splice(2).
> +
> blind-transform affine transformation by matrix multiplication, -[xy] for
> tiling, -s for
> improve quality on downscaling (pixels' neighbours must not change)
> blind-apply-map remap pixels (distortion) using the X and Y values, -[xy]
> for tiling, -s for
>
>
Received on Tue Dec 12 2017 - 12:02:11 CET