gmane.comp.file-systems.ext3.user

Re: Write ordering in Ext4

Theodore Ts'o — 2013-06-04T19:08:59

The fact that you are asking all of these questions is making me very
nervous.  Why do you care?  Application programmers should ***not***
be depending on low-level file system behavior.

If you care about what might happen after a crash, you need to use
fsync().

- Ted

Re: Write ordering in Ext4

Eric Sandeen — 2013-06-04T17:33:25

If you mean turn off delayed allocation, look no further than the mount
options documented in the kernel tree, Documentation/filesystems/ext4.txt:

nodelalloc              Disable delayed allocation.  Blocks are allocated
                        when the data is copied from userspace to the
                        page cache, either via the write(2) system call
                        or when an mmap'ed page which was previously
                        unallocated is written for the first time.

Out of curiosity, why do you want to turn off delalloc?

-Eric

Re: Write ordering in Ext4

Arul Selvan — 2013-06-04T17:17:38

thanks that answered my question. One more question, is it possible to stop the delayed block allocation in ext4 ?

On 2013-06-02, at 23:33, "Arul Selvan" <Rarul< at >novell.com> wrote:

Writes in memory to the same file are serialized by i_mutex, but may
modify the same page in memory repeatedly.

When that page us being written to disk, it will be marked with the
page writeback flag, in order to stabilize the content, and allow consistent
checksums (e.g. for MD RAID or disks with T10-DIF). This may block
any further writes from modifying the same page as it is being
submitted to disk, depending on the kernel version and the
requirements of the underlying storage. Once the disk write has been
finished, the writeback bit is cleared and the page can be modified again.

In all cases, the writes to a single page are ordered, but there is no
_guarantee_ about writes to different data blocks being ordered.
The ext4 journal will in fact impose some order on data writes,
by ensuring that the data from all writes associa

Re: Write ordering in Ext4

Andreas Dilger — 2013-06-03T14:47:38

Writes in memory to the same file are serialized by i_mutex, but may
modify the same page in memory repeatedly.

When that page us being written to disk, it will be marked with the
page writeback flag, in order to stabilize the content, and allow consistent
checksums (e.g. for MD RAID or disks with T10-DIF). This may block
any further writes from modifying the same page as it is being
submitted to disk, depending on the kernel version and the
requirements of the underlying storage. Once the disk write has been
finished, the writeback bit is cleared and the page can be modified again.

In all cases, the writes to a single page are ordered, but there is no
_guarantee_ about writes to different data blocks being ordered.
The ext4 journal will in fact impose some order on data writes,
by ensuring that the data from all writes associated with a transaction
are flushed before the data for the next transaction.

Since fsync() of any file commits the current transaction, this has
the side-effect that any fsync caus

Write ordering in Ext4

Arul Selvan — 2013-06-03T05:33:39

Greetings. I am Arul Selvan works for Novell. I am exploring the Ext4 architecture, more specifically i would like to understand the write ordering, basically the same blocks is modified more than once, how the write is ordered. Could you point me the doc or the specific source file to look.
_______________________________________________
Ext3-users mailing list
Ext3-users< at >redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Re: (LONG) Delay when writing to ext4 LVM after boot

Andreas Dilger — 2013-04-24T15:58:21

This is a problem that I am very familiar with for large filesystems. The issue is that if the filesystem is relatively full, the first write needs to load and search a lot of the block bitmaps to try and find enough space to allocate blocks for the write.  Depending on how it was formatted, each block bitmap read needs a seek. 


Might I guess that this filesystem was formatted as ext3 and not as ext4?  In particular, is the "flex_bg" option missing from the Features line in the "dumpe2fs -h /dev/XXX" output?  This feature is enabled by default if formatting as ext4, but not as ext3.

The flex_bg feature will allocate the block bitmaps in large chunks on the disk so that they can be loaded quickly at mount and e2fsck time.  On a 16TB filesystem with 10 ms seek time, in the worst case without flex_bg it could take up to 20 minutes to load all of the bitmaps at boot time without flex_bg...


Unfortunately, flex_bg is a format-time option, so you would need a full backup-restore to benefit from it for your fi

(LONG) Delay when writing to ext4 LVM after boot

Ken Bass — 2013-04-24T00:14:52

(I previously asked this question in the LVM list, and they suggested I ask
here.)

I have a large LV, about 6.5T, consisting of 4 physical drives of various
sizes. The LV is formatted as ext4. There is no raid  involved (hardware of
software).

After I first boot, if I try to write a large file (>~ 80M) to this LV, the
write hangs for about 1minute or more, then continues on at full speed and
finishes successfully. Writes of small files don't show this delay. After
that first write and delay, all subsequent writes to other large files
proceed at full speed.

I am currently running Fedora 17 64bit  (kernel 3.8.4-102.fc17.x86_64) but
have noticed this also in previous systems (both 64 and 32bit). With
smaller file systems ( < 1T ), there was a delay, but it was small, and it
increased significantly as I increased the LV size.

I have run e2fsck with the -D option (before attempting a write), which
made no difference. Also, fwiw,  I am mounting this with the default
options. I've tried other options that were

Re: e2freefrag says filesystem too large

Theodore Ts'o — 2013-03-27T02:14:43

Yes, it's fixed in e2fsprogs 1.42.7.  And if you're using 64-bit file
systems, you really really want to upgrade to 1.42.7 --- especially if
you are ever thinking of using resize2fs; we fixed a number of very
serious bugs, some of which could destroy file systems if you try to
do off-line resizes.  See the release notes for more details:

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42.7

Regards,

- Ted

e2freefrag says filesystem too large

2013-03-26T19:51:24


Can someone tell me if this will be fixed?

# e2freefrag
/dev/sdl1
Device: /dev/sdl1
Blocksize: 4096 bytes
/dev/sdl1: Filesystem too large to use legacy bitmaps while reading
block bitmap

# rpm -qa|grep e2fsprogs
e2fsprogs-libs-1.42.5-1.fc18.x86_64
e2fsprogs-1.42.5-1.fc18.x86_64

# df|grep /dev/sdl1
/dev/sdl1Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 
35143869536 30265426892Â  4175317880Â  88% /mnt/backup

Thanks!

---
Will Y.

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-24T22:40:56

  I did not find it in  git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git

  There are some references from Lustre doc [1] and I could find some
github repo with some code [2]. Would you know where the e2scan upstream
lives ?


  [1]
http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_55923
  [2] https://github.com/morrone/e2fsprogs

Re: ext4 and extremely slow filesystem traversal

Bodo Thiesen — 2013-03-16T11:29:07

* Vincent Caron <vcaron< at >bearstech.com> hat geschrieben:


Using -m technically makes the file system driver report ENOSPC when
there is in fact still free space available.

So, as long as you make sure, that you have at least 5% free space at any
given time, it doesn't matter whether you have -m0 or -m5. However, tools
like df show the available capacity to user space, so 100GB with 95GB used
will show 100% used with -m5 and 95% used with -m0. Effectively, that
means, when you go with -m5 and make sure, df shows at least 5% free space,
you end up with about 10% free all the time - this way reducing
fragmentation - in theory. Since you're missusing your file system as
data bank management system with many small files anyways,
inter file fragmentation is the least of your problems. So, it's totally
safe for you to stay with -m0.

Regards, Bodo

Re: ext4 and extremely slow filesystem traversal

Bodo Thiesen — 2013-03-16T11:17:01

* Vincent Caron <vcaron< at >bearstech.com> hat geschrieben:


There is a simple and reliable solution for block level backups: dd

umount && dd if="our raid partition" of="some new big enough disc" &&
mount

and then wait for the data to go at 100MB/s or so to the new disc.

Using snapshots is not a reliable way to do backups, since you would
still have to trust the LVM code to be totally error free and protect
your data under any circumstances (including hardware failures in your
raid array etc).

For your actual problem: Ask your developers to use some mapping
system.

When they want to access a file "filename" then calculate md5sum of
"filename", take the first 6 characters of the ascii representation
(here it would be 435ed7) and create a file called "43/5e/d7-X".
This way you would end up with at most 65792 directories. The X is needed
to distinquish between files with same 6 first letters md5sum. So, first
file gets name "43/5e/d7-1", second file gets name "43/5e/d7-2" and so on.

Somewhere else, they woul

Re: ext4 and extremely slow filesystem traversal

Andreas Dilger — 2013-03-15T07:14:17


That's exactly what e2scan does. I'm pretty sure that is in upstream e2fsprogs now (not just our Lustre version), but I'm on a plane and cannot check.

It will scan the inode table directly and can generate the pathnames of files efficiently. It can filter on timestamps. 

Cheers, Andreas

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-14T23:07:31

  Very useful. Global stats without having to scan the whole filesystems
are very precious...

  I was wondering : couldn't we use dumpe2fs or something based on
libext2fs to quickly extract a snapshot of all inodes from a given
filesystem ? For incremental backups, simply checking the mtime on
millions of inodes and discovering that only a handful of them were
updated since the previous pass looks very inefficient with
readdir()+lstat(). So mnay syscalls, so man spoonfed bits of
information. When I had a peek, I tought I'd got a list of inodes but
would not be able to link them back to their name(s) without inducing
the same cost as a regular find-like filesystem traversal. Does it make
sense ?

  AFAIK I would be better served with block-level snapshot solutions,
but LVM snapshots are supposed to double your writes if I got it right,
and I'm not sure there's something else in the Linux and free software
world. Plus I'd love to not migrate away from my ext3/4's without a
compelling reason. Btrfs is not (ye

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-14T22:56:47

  Because the man page says that reserved blocks are used to protect
root-level daemons from misbehaving would unprivileged programs try to
fill the disk. And uh, to avoid fragmentation, I missed that part.

  OTOH I monitor disk space and never let go past 95% block usage
without specific action (freeing inodes or enlarging filesystem). Would
I use -m5 and oversize my filesystems (because I sell the capacity, say
I sell 100GB then I need a 105GB blockdev), I would still monitor the
disk usage and take action before it's 100% filled up. But I'd end up
reserving more blocks without more guarantees that the -m0 case.

  So technically it looks wrong, but politically I'm not sure it's
stupid. Or is it ?

Re: ext4 and extremely slow filesystem traversal

Peter Grandi — 2013-03-14T20:57:49



Why does it matter here that "no file owned by root"?

What has that got to do with the much greater difficulty to find
contiguous space the fuller the filetree is?

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-14T01:05:51

Just as a note, e2fsck -v can sometimes get this information much more
quickly than other alternatives, since it can scan the file system in
inode order, instead of the essentially random order.

Just as a side, if you just want to get a rough count of the number of
directories, you can get that by grabbing the information out of
dumpe2fs.

Group 624: (Blocks 20447232-20479999) [ITABLE_ZEROED]
  Checksum 0xd3f5, unused inodes 4821
  Block bitmap at 20447232 (+0), Inode bitmap at 20447248 (+16)
  Inode table at 20447264-20447775 (+32)
  24103 free blocks, 4821 free inodes, 435 directories, 4821 unused inodes
                                       ^^^^^^^^^^^^^^^
  Free blocks: 20455889, 20455898-20479999
  Free inodes: 5115180-5120000

Dumpe2fs doesn't actually sum the number of directories, and you won't
be able to differentiate the number of files that are regular files versus
symlinks, device nodes, etc., but if you just want to get the number
of directories, you can get this number by getting the informa

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-13T21:50:13

  This filesystem has no file owned by root and won't have any. I
thought in this case -m0 would be a good idea.

  Thanks a lot for your detailed insight on the various performance
figures, I didn't do the proper math to realize that this inode reading
rate was actually *good*.

  Fortunately the client is technically savvy, and pointing him at this
mailing-list thread will help make him the right decision.

Re: ext4 and extremely slow filesystem traversal

Peter Grandi — 2013-03-13T21:29:32

That is roughly 1M inodes/hour and 20GB/hour, or  nearly 300
inodes/s and nearly 6MB/s. These are very good numbers for high
random IOPS loads, and as seen later, you have one.


That helps.


That's the pointless default... But does not particularly slow
things down here.


The striping or alignment are not relevant on reads, but the
stride matters a great deal as to metadata parallelism, and here
it is set to 64KiB. But the array stride is 16KiB (a 4-wide
stripe of 64KiB). But since it is an integral multiple it should
be about as good. And since the backup performance is pretty
good, that seems the case.


That's usually a truly terrible setting (20% is a much better
value), but your filesystem is not very full anyhow.


Rather they are pretty good. Each 500GB SATA disk can usually do
somewhat less than 100 random IOPS/second, there are 4 disks in
each stripe when reading, and you are getting nearly 300 inodes/s
and 5MB/s, quite close to the maximum. On random loads with
smallish records typical rotating

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-13T20:33:46

Wow.  You have more directories than regular files!  Given that there
are no hard links, that implies that you have at least 2,079,271
directories which are ***empty***.

The inline data feature (which is still in testing and isn't something
I can recommend for production use yet) is probably the best hope for
you.  But probably the best thing you can do is to harrague your
developers to ask what the heck they are doing....

                - Ted

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-13T20:52:10

To be clear, that's at least two million directories assuming that all
of the other directories have but a single file in them(!).  In reality
you probably have a lot more than 2 million empty directories....

                    - Ted