On Sun, Apr 21, 2019 at 8:42 PM Daniel Cegiełka
<daniel.cegielka_AT_gmail.com> wrote:
> We have three main purpose filesystems:
> * reading - Speed is the key here.
> * writing - This is a very complex issue. Write operations should be
> atomic (see Raiser4). Data integrity should be checked at the memory
> level. And of course speed.
> * storage - Speed is not the most important factor here. The key is
> data integrity and their safe storage.
You are missing anonther important use-case: database storage -- a
file-system that is tailored to large files, low in number, and where
file churn (i.e. creation and deletion) is almost zero.
And another important use-case: archival storage -- append-only
file-system that is tailored to storing backup data. This is somewhat
different from your "storage" use-case, as in my view an archival
storage should be append-only, thus immutable for the already written
portions.
> The best example here is Google. They used ext2 for a long time as
> their main filesystem. Yes, ext2. Why? ext2 is a very simple file
> system - it is light and fast. One of the best fs if fast data reading
> is important. So no ZFS, no btrfs etc.
Can you provide a reference for Ext2 beeing used extensively by
Google? (I don't question this statement, I just want to read more
about it.)
Regarding Ext2, I thought about using it for a backup file-system
targeted at storing large files, for archival purposes. My reasoning
was this: regardless of what OS I will end up using 20 years from
now, or if the storage medium has some blocks damaged (which might be
part of the actual file-system meta-data), I assume Ext2 is quite easy
to implement by hand so I can try to recover something. Moreover, the
simplest the code, the less chance for something going wrong and
deleting my data.
> Ok, but what about the data integrity? An example here is Miscrosoft
> and GVFS Driver. They turned git into a virtual filesystem. Git
> provides data integrity checks, backup and cloning of data to other
> machines. See Michael Forney's Oasis distro. He uses git to manage the
> system:
Indeed, Git is the best "file-system" (if it can be called so) for
important data. (It even has `git-fsck`.) :)
Moreover I usually keep (and generate once in a while) MD5 checksums
of all my files, which, coupled with a backup strategy, could provide
a way to recover data even long after it was initially created. These
MD5 files can also be used to "diff" the contents at various points in
time.
(I know that MD5 isn't "cryptographically strong" anymore, however I
don't need that feature, I only need "good enough" fingerprinting.)
> "ext4, which works - mostly - but is showing its age. The codebase
> terrifies most filesystem developers who have had to work on it, and
> heavy users still run into terrifying performance and data corruption
> bugs with frightening regularity. The general opinion of filesystem
> developers is that it's a miracle it works as well as it does, and
> ext4's best feature is its fsck (which does indeed work miracles)."
Indeed once in a while there is a nasty bug that can "loose data"...
Here is for example an article from LWN.net from last year about this
topic:
https://lwn.net/Articles/774440/
> Summing up: JSF seems to me an interesting alternative to ext4. I know
> that JFS has a lot of shortcomings, and will require a lot of work and
> patches.
I don't know if I personally would be able to trust another
non-mainstream file-system for my storage purposes (I've started with
ReiserFS-3, and was burned, then moved to JFS and was almost
burned)... I think the best outcome for everyone, is if those capable
of developing file-systems would focus on a handfull implementations
and perfect them.
For example at the beginning of this year I've played somewhat with
Ext4, and I think the most important feature it is missing is a
thorough documentation regarding how to use it's miriad of options and
features... I've dwelved somewhat into all its options but it was
quite hard, and I still don't trust I made the best choice...
Ciprian.
Received on Sun Apr 21 2019 - 21:52:55 CEST