RE: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

From: Mattias Andrée <maandree_AT_kth.se>
Date: Tue, 12 Dec 2017 14:49:34 +0000

It may look insane on the surface level. I am not commited to changes
listed in the TODO, most of them are just ideas that I will have to evalute
later whether they are worth implementing. Although I'm pretty convinced
most of them are good.
________________________________________
From: isabella parakiss [izaberina_AT_gmail.com]
Sent: 12 December 2017 13:31
To: hackers mail list
Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

don't you realize how insane the whole thing is?

On 12/12/17, Mattias Andrée <maandree_AT_kth.se> wrote:
> Perhaps I should clarify that (1) the goal would be to have blind-tee
> (and blind-cat if that is implemented) to use already existing functions
> that sends data between two files, and have these functions use splice
> when possible), so would only use tee explicitly, not splice, and (2)
> a 1 hour long blind video with the resolution 1920x1080_AT_30 is 6.5TB
> large, so we are talking about a massive amount of data that is beeing
> sent between processes. In total, a 1 hour long video could require
> houndreds of terabytes being sent between process. Double that if it is a
> 60 fps video (which you will actually on the Internet nowadays) and
> multiple that by 4 for a 4K video, and double that again for a stereoscopic
> video (you can find all of these things on Blu-ray videos). So we could
> be talking about a couple of petabytes of data for a profession video,
> and 100TB for an amateur video, not just a couple of gigabytes. Try to
> copy that about of data with dd (in parallell of course), and you might
> find that even halving that time[1] would be nice, even if most if the time
> in the rendering process is usually spent on rendering effects, stacking
> videos, and transcoding.
>
> [1] 42 hours per petabyte.
>
> ________________________________________
> From: Mattias Andrée [maandree_AT_kth.se]
> Sent: 12 December 2017 11:32
> To: hackers mail list
> Subject: RE: [hackers] [blind] update todo: tee is too slow || Mattias
> Andrée
>
> When I rendered a video, tee used 100% while the other process was
> basically at 0, for more than 50% of the rendering time. I ran it multiply
> times to verify that is was correct. An alternative solution would be
> to use sockets, but that would require changes to the shell. Optimising
> tee seems like the sensible alternative. Besides, normally when you use
> tools like tee and cat, you expect it to finish within milliseconds, even
> for larger files, here we are talking about time intervals between seconds
> and a few hours, with most if not all CPU:s at 90% to 100%, so
> optimisations
> do not hurt, especially not when as significant as using tee and splice. It
> may be unfortunate to have to use both tee–splice and read–write (which
> is required both for platforms not supporting tee and splice, and for file
> types not supporting them), but considering how little complexity this
> adds,
> — it's not much more than a duplication of a function, — I think it is a
> trade-off worth considering.
>
> ________________________________________
> From: isabella parakiss [izaberina_AT_gmail.com]
> Sent: 12 December 2017 10:58
> To: hackers mail list
> Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias
> Andrée
>
> bullshit
>
> On 12/4/17, git_AT_suckless.org <git_AT_suckless.org> wrote:
>> commit d8aa45da86d1128149fd7ab6ac3725bf8e88a1b1
>> Author: Mattias Andrée <maandree_AT_kth.se>
>> AuthorDate: Mon Dec 4 22:35:59 2017 +0100
>> Commit: Mattias Andrée <maandree_AT_kth.se>
>> CommitDate: Mon Dec 4 22:35:59 2017 +0100
>>
>> update todo: tee is too slow
>>
>> Signed-off-by: Mattias Andrée <maandree_AT_kth.se>
>>
>> diff --git a/TODO b/TODO
>> index b0bbde7..408e942 100644
>> --- a/TODO
>> +++ b/TODO
>> _AT_@ -1,3 +1,7 @@
>> +blind-tee (and tee(1)) is too slow (bottleneck) and must be
>> reimplemented
>> +using tee(2) and splice(2). cat(1) may also be too slow, if this is the
>> +case, add blind-splice that just copies stdin to stdout using splice(2).
>> +
>> blind-transform affine transformation by matrix
>> multiplication, -[xy] for
>> tiling, -s for
>> improve quality on downscaling (pixels'
>> neighbours must not change)
>> blind-apply-map remap pixels (distortion) using the X and Y
>> values, -[xy]
>> for tiling, -s for
>>
>>
>
>
>
>
Received on Tue Dec 12 2017 - 15:49:34 CET