Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée

From: Mattias Andrée <maandree_AT_kth.se>
Date: Wed, 13 Dec 2017 08:31:21 +0100

I'm not convinced, but since you guys disagree I will not implement this.

On Wed, 13 Dec 2017 00:45:18 +0000
Richard Ipsum <richardipsum_AT_fastmail.co.uk> wrote:

> I lack community standing to really comment on this.
>
> That being said, as a casual observer it seems fairly obvious to me
> that what you're proposing is totally anathema to suckless philosophy.
>
> On Tue, Dec 12, 2017 at 02:49:34PM +0000, Mattias Andrée wrote:
> > It may look insane on the surface level. I am not commited to changes
> > listed in the TODO, most of them are just ideas that I will have to evalute
> > later whether they are worth implementing. Although I'm pretty convinced
> > most of them are good.
> > ________________________________________
> > From: isabella parakiss [izaberina_AT_gmail.com]
> > Sent: 12 December 2017 13:31
> > To: hackers mail list
> > Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée
> >
> > don't you realize how insane the whole thing is?
> >
> > On 12/12/17, Mattias Andrée <maandree_AT_kth.se> wrote:
> > > Perhaps I should clarify that (1) the goal would be to have blind-tee
> > > (and blind-cat if that is implemented) to use already existing functions
> > > that sends data between two files, and have these functions use splice
> > > when possible), so would only use tee explicitly, not splice, and (2)
> > > a 1 hour long blind video with the resolution 1920x1080_AT_30 is 6.5TB
> > > large, so we are talking about a massive amount of data that is beeing
> > > sent between processes. In total, a 1 hour long video could require
> > > houndreds of terabytes being sent between process. Double that if it is a
> > > 60 fps video (which you will actually on the Internet nowadays) and
> > > multiple that by 4 for a 4K video, and double that again for a stereoscopic
> > > video (you can find all of these things on Blu-ray videos). So we could
> > > be talking about a couple of petabytes of data for a profession video,
> > > and 100TB for an amateur video, not just a couple of gigabytes. Try to
> > > copy that about of data with dd (in parallell of course), and you might
> > > find that even halving that time[1] would be nice, even if most if the time
> > > in the rendering process is usually spent on rendering effects, stacking
> > > videos, and transcoding.
> > >
> > > [1] 42 hours per petabyte.
> > >
> > > ________________________________________
> > > From: Mattias Andrée [maandree_AT_kth.se]
> > > Sent: 12 December 2017 11:32
> > > To: hackers mail list
> > > Subject: RE: [hackers] [blind] update todo: tee is too slow || Mattias
> > > Andrée
> > >
> > > When I rendered a video, tee used 100% while the other process was
> > > basically at 0, for more than 50% of the rendering time. I ran it multiply
> > > times to verify that is was correct. An alternative solution would be
> > > to use sockets, but that would require changes to the shell. Optimising
> > > tee seems like the sensible alternative. Besides, normally when you use
> > > tools like tee and cat, you expect it to finish within milliseconds, even
> > > for larger files, here we are talking about time intervals between seconds
> > > and a few hours, with most if not all CPU:s at 90% to 100%, so
> > > optimisations
> > > do not hurt, especially not when as significant as using tee and splice. It
> > > may be unfortunate to have to use both tee–splice and read–write (which
> > > is required both for platforms not supporting tee and splice, and for file
> > > types not supporting them), but considering how little complexity this
> > > adds,
> > > — it's not much more than a duplication of a function, — I think it is a
> > > trade-off worth considering.
> > >
> > > ________________________________________
> > > From: isabella parakiss [izaberina_AT_gmail.com]
> > > Sent: 12 December 2017 10:58
> > > To: hackers mail list
> > > Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias
> > > Andrée
> > >
> > > bullshit
> > >
> > > On 12/4/17, git_AT_suckless.org <git_AT_suckless.org> wrote:
> > >> commit d8aa45da86d1128149fd7ab6ac3725bf8e88a1b1
> > >> Author: Mattias Andrée <maandree_AT_kth.se>
> > >> AuthorDate: Mon Dec 4 22:35:59 2017 +0100
> > >> Commit: Mattias Andrée <maandree_AT_kth.se>
> > >> CommitDate: Mon Dec 4 22:35:59 2017 +0100
> > >>
> > >> update todo: tee is too slow
> > >>
> > >> Signed-off-by: Mattias Andrée <maandree_AT_kth.se>
> > >>
> > >> diff --git a/TODO b/TODO
> > >> index b0bbde7..408e942 100644
> > >> --- a/TODO
> > >> +++ b/TODO
> > >> _AT_@ -1,3 +1,7 @@
> > >> +blind-tee (and tee(1)) is too slow (bottleneck) and must be
> > >> reimplemented
> > >> +using tee(2) and splice(2). cat(1) may also be too slow, if this is the
> > >> +case, add blind-splice that just copies stdin to stdout using splice(2).
> > >> +
> > >> blind-transform affine transformation by matrix
> > >> multiplication, -[xy] for
> > >> tiling, -s for
> > >> improve quality on downscaling (pixels'
> > >> neighbours must not change)
> > >> blind-apply-map remap pixels (distortion) using the X and Y
> > >> values, -[xy]
> > >> for tiling, -s for
> > >>
> > >>
> > >
> > >
> > >
> > >
> >
> >
>


Received on Wed Dec 13 2017 - 08:31:21 CET

This archive was generated by hypermail 2.3.0 : Wed Dec 13 2017 - 08:36:29 CET