Re: [dev] [sbase] sponge v2

From: Bjartur Thorlacius <svartman95_AT_gmail.com>
Date: Wed, 03 Jul 2013 20:35:16 +0000

On 07/02/2013 07:54 PM, Calvin Morrison wrote:
>> >If on any system other than linux, I would consider loading into ram,
>> >but because of memory overcommit, malloc never fails, the whole system
>> >crawls to a halt, and the oom killer takes 20 minutes to put
>> >everything back together. No thanks.
> Okay so you are discussing a problem with your version of unix and
> malloc, not with the tool sponge.
This being sbase, the discussion should boil down to whether it's
simpler to malloc or write and read. Writing data to a tempfile marks it
for paging. If there's a simpler way to do that, or if that is in fact
unwanted, tempfiles should be eschewed.

You're just storing data into memory. Just tell the OS what you're
doing, hand over the data and be done with it. The problem with
mallocing large amounts of data is that malloc'd data is byte
addressable, implying that for O_LARGEFILEs you need practically 64bit
addresses, which on current machines can lead to (minor) performance
penalties. This can be worked around simply by telling the OS that you
don't need to be able to address all of the data at once: buffered write()s.

So, if you ever need to pipe a multi-gibibyte file through a shell
command on a 32bit machine, you want sponge to use tempfiles. Otherwise,
you shouldn't care.
Received on Wed Jul 03 2013 - 22:35:16 CEST