Re: [dev] [sbase] sponge v2

From: hiro <23hiro_AT_gmail.com>
Date: Wed, 3 Jul 2013 23:01:30 +0200

just give it up, this thread sucks.

On 7/3/13, Bjartur Thorlacius <svartman95_AT_gmail.com> wrote:
> On 07/02/2013 07:54 PM, Calvin Morrison wrote:
>>> >If on any system other than linux, I would consider loading into ram,
>>> >but because of memory overcommit, malloc never fails, the whole system
>>> >crawls to a halt, and the oom killer takes 20 minutes to put
>>> >everything back together. No thanks.
>> Okay so you are discussing a problem with your version of unix and
>> malloc, not with the tool sponge.
> This being sbase, the discussion should boil down to whether it's
> simpler to malloc or write and read. Writing data to a tempfile marks it
> for paging. If there's a simpler way to do that, or if that is in fact
> unwanted, tempfiles should be eschewed.
>
> You're just storing data into memory. Just tell the OS what you're
> doing, hand over the data and be done with it. The problem with
> mallocing large amounts of data is that malloc'd data is byte
> addressable, implying that for O_LARGEFILEs you need practically 64bit
> addresses, which on current machines can lead to (minor) performance
> penalties. This can be worked around simply by telling the OS that you
> don't need to be able to address all of the data at once: buffered
> write()s.
>
> So, if you ever need to pipe a multi-gibibyte file through a shell
> command on a 32bit machine, you want sponge to use tempfiles. Otherwise,
> you shouldn't care.
>
>
Received on Wed Jul 03 2013 - 23:01:30 CEST