gmane.comp.file-systems.ext3.user

Re: (LONG) Delay when writing to ext4 LVM after boot

Andreas Dilger — 2013-04-24T15:58:21

This is a problem that I am very familiar with for large filesystems. The issue is that if the filesystem is relatively full, the first write needs to load and search a lot of the block bitmaps to try and find enough space to allocate blocks for the write.  Depending on how it was formatted, each block bitmap read needs a seek. 


Might I guess that this filesystem was formatted as ext3 and not as ext4?  In particular, is the "flex_bg" option missing from the Features line in the "dumpe2fs -h /dev/XXX" output?  This feature is enabled by default if formatting as ext4, but not as ext3.

The flex_bg feature will allocate the block bitmaps in large chunks on the disk so that they can be loaded quickly at mount and e2fsck time.  On a 16TB filesystem with 10 ms seek time, in the worst case without flex_bg it could take up to 20 minutes to load all of the bitmaps at boot time without flex_bg...


Unfortunately, flex_bg is a format-time option, so you would need a full backup-restore to benefit from it for your fi

(LONG) Delay when writing to ext4 LVM after boot

Ken Bass — 2013-04-24T00:14:52

(I previously asked this question in the LVM list, and they suggested I ask
here.)

I have a large LV, about 6.5T, consisting of 4 physical drives of various
sizes. The LV is formatted as ext4. There is no raid  involved (hardware of
software).

After I first boot, if I try to write a large file (>~ 80M) to this LV, the
write hangs for about 1minute or more, then continues on at full speed and
finishes successfully. Writes of small files don't show this delay. After
that first write and delay, all subsequent writes to other large files
proceed at full speed.

I am currently running Fedora 17 64bit  (kernel 3.8.4-102.fc17.x86_64) but
have noticed this also in previous systems (both 64 and 32bit). With
smaller file systems ( < 1T ), there was a delay, but it was small, and it
increased significantly as I increased the LV size.

I have run e2fsck with the -D option (before attempting a write), which
made no difference. Also, fwiw,  I am mounting this with the default
options. I've tried other options that were

Re: e2freefrag says filesystem too large

Theodore Ts'o — 2013-03-27T02:14:43

Yes, it's fixed in e2fsprogs 1.42.7.  And if you're using 64-bit file
systems, you really really want to upgrade to 1.42.7 --- especially if
you are ever thinking of using resize2fs; we fixed a number of very
serious bugs, some of which could destroy file systems if you try to
do off-line resizes.  See the release notes for more details:

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42.7

Regards,

- Ted

e2freefrag says filesystem too large

2013-03-26T19:51:24


Can someone tell me if this will be fixed?

# e2freefrag
/dev/sdl1
Device: /dev/sdl1
Blocksize: 4096 bytes
/dev/sdl1: Filesystem too large to use legacy bitmaps while reading
block bitmap

# rpm -qa|grep e2fsprogs
e2fsprogs-libs-1.42.5-1.fc18.x86_64
e2fsprogs-1.42.5-1.fc18.x86_64

# df|grep /dev/sdl1
/dev/sdl1Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 
35143869536 30265426892Â  4175317880Â  88% /mnt/backup

Thanks!

---
Will Y.

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-24T22:40:56

  I did not find it in  git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git

  There are some references from Lustre doc [1] and I could find some
github repo with some code [2]. Would you know where the e2scan upstream
lives ?


  [1]
http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_55923
  [2] https://github.com/morrone/e2fsprogs

Re: ext4 and extremely slow filesystem traversal

Bodo Thiesen — 2013-03-16T11:29:07

* Vincent Caron <vcaron< at >bearstech.com> hat geschrieben:


Using -m technically makes the file system driver report ENOSPC when
there is in fact still free space available.

So, as long as you make sure, that you have at least 5% free space at any
given time, it doesn't matter whether you have -m0 or -m5. However, tools
like df show the available capacity to user space, so 100GB with 95GB used
will show 100% used with -m5 and 95% used with -m0. Effectively, that
means, when you go with -m5 and make sure, df shows at least 5% free space,
you end up with about 10% free all the time - this way reducing
fragmentation - in theory. Since you're missusing your file system as
data bank management system with many small files anyways,
inter file fragmentation is the least of your problems. So, it's totally
safe for you to stay with -m0.

Regards, Bodo

Re: ext4 and extremely slow filesystem traversal

Bodo Thiesen — 2013-03-16T11:17:01

* Vincent Caron <vcaron< at >bearstech.com> hat geschrieben:


There is a simple and reliable solution for block level backups: dd

umount && dd if="our raid partition" of="some new big enough disc" &&
mount

and then wait for the data to go at 100MB/s or so to the new disc.

Using snapshots is not a reliable way to do backups, since you would
still have to trust the LVM code to be totally error free and protect
your data under any circumstances (including hardware failures in your
raid array etc).

For your actual problem: Ask your developers to use some mapping
system.

When they want to access a file "filename" then calculate md5sum of
"filename", take the first 6 characters of the ascii representation
(here it would be 435ed7) and create a file called "43/5e/d7-X".
This way you would end up with at most 65792 directories. The X is needed
to distinquish between files with same 6 first letters md5sum. So, first
file gets name "43/5e/d7-1", second file gets name "43/5e/d7-2" and so on.

Somewhere else, they woul

Re: ext4 and extremely slow filesystem traversal

Andreas Dilger — 2013-03-15T07:14:17


That's exactly what e2scan does. I'm pretty sure that is in upstream e2fsprogs now (not just our Lustre version), but I'm on a plane and cannot check.

It will scan the inode table directly and can generate the pathnames of files efficiently. It can filter on timestamps. 

Cheers, Andreas

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-14T23:07:31

  Very useful. Global stats without having to scan the whole filesystems
are very precious...

  I was wondering : couldn't we use dumpe2fs or something based on
libext2fs to quickly extract a snapshot of all inodes from a given
filesystem ? For incremental backups, simply checking the mtime on
millions of inodes and discovering that only a handful of them were
updated since the previous pass looks very inefficient with
readdir()+lstat(). So mnay syscalls, so man spoonfed bits of
information. When I had a peek, I tought I'd got a list of inodes but
would not be able to link them back to their name(s) without inducing
the same cost as a regular find-like filesystem traversal. Does it make
sense ?

  AFAIK I would be better served with block-level snapshot solutions,
but LVM snapshots are supposed to double your writes if I got it right,
and I'm not sure there's something else in the Linux and free software
world. Plus I'd love to not migrate away from my ext3/4's without a
compelling reason. Btrfs is not (ye

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-14T22:56:47

  Because the man page says that reserved blocks are used to protect
root-level daemons from misbehaving would unprivileged programs try to
fill the disk. And uh, to avoid fragmentation, I missed that part.

  OTOH I monitor disk space and never let go past 95% block usage
without specific action (freeing inodes or enlarging filesystem). Would
I use -m5 and oversize my filesystems (because I sell the capacity, say
I sell 100GB then I need a 105GB blockdev), I would still monitor the
disk usage and take action before it's 100% filled up. But I'd end up
reserving more blocks without more guarantees that the -m0 case.

  So technically it looks wrong, but politically I'm not sure it's
stupid. Or is it ?

Re: ext4 and extremely slow filesystem traversal

Peter Grandi — 2013-03-14T20:57:49



Why does it matter here that "no file owned by root"?

What has that got to do with the much greater difficulty to find
contiguous space the fuller the filetree is?

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-14T01:05:51

Just as a note, e2fsck -v can sometimes get this information much more
quickly than other alternatives, since it can scan the file system in
inode order, instead of the essentially random order.

Just as a side, if you just want to get a rough count of the number of
directories, you can get that by grabbing the information out of
dumpe2fs.

Group 624: (Blocks 20447232-20479999) [ITABLE_ZEROED]
  Checksum 0xd3f5, unused inodes 4821
  Block bitmap at 20447232 (+0), Inode bitmap at 20447248 (+16)
  Inode table at 20447264-20447775 (+32)
  24103 free blocks, 4821 free inodes, 435 directories, 4821 unused inodes
                                       ^^^^^^^^^^^^^^^
  Free blocks: 20455889, 20455898-20479999
  Free inodes: 5115180-5120000

Dumpe2fs doesn't actually sum the number of directories, and you won't
be able to differentiate the number of files that are regular files versus
symlinks, device nodes, etc., but if you just want to get the number
of directories, you can get this number by getting the informa

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-13T21:50:13

  This filesystem has no file owned by root and won't have any. I
thought in this case -m0 would be a good idea.

  Thanks a lot for your detailed insight on the various performance
figures, I didn't do the proper math to realize that this inode reading
rate was actually *good*.

  Fortunately the client is technically savvy, and pointing him at this
mailing-list thread will help make him the right decision.

Re: ext4 and extremely slow filesystem traversal

Peter Grandi — 2013-03-13T21:29:32

That is roughly 1M inodes/hour and 20GB/hour, or  nearly 300
inodes/s and nearly 6MB/s. These are very good numbers for high
random IOPS loads, and as seen later, you have one.


That helps.


That's the pointless default... But does not particularly slow
things down here.


The striping or alignment are not relevant on reads, but the
stride matters a great deal as to metadata parallelism, and here
it is set to 64KiB. But the array stride is 16KiB (a 4-wide
stripe of 64KiB). But since it is an integral multiple it should
be about as good. And since the backup performance is pretty
good, that seems the case.


That's usually a truly terrible setting (20% is a much better
value), but your filesystem is not very full anyhow.


Rather they are pretty good. Each 500GB SATA disk can usually do
somewhat less than 100 random IOPS/second, there are 4 disks in
each stripe when reading, and you are getting nearly 300 inodes/s
and 5MB/s, quite close to the maximum. On random loads with
smallish records typical rotating

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-13T20:33:46

Wow.  You have more directories than regular files!  Given that there
are no hard links, that implies that you have at least 2,079,271
directories which are ***empty***.

The inline data feature (which is still in testing and isn't something
I can recommend for production use yet) is probably the best hope for
you.  But probably the best thing you can do is to harrague your
developers to ask what the heck they are doing....

                - Ted

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-13T20:52:10

To be clear, that's at least two million directories assuming that all
of the other directories have but a single file in them(!).  In reality
you probably have a lot more than 2 million empty directories....

                    - Ted

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-13T20:49:20

  Awful, isn't it ? I knew directories were abused, but didn't know that
'e2fsck -v' would display the exact figures (since I never waited 5+
hours to scan the whole filesystem). Nice to know.



  Indeed, these filers are storing live and sensitive data and are
conservatively running stable OS and well known kernels.

  Thanks for your advice, I'll actively work with the devs in order to
refactor their filesystem layout.

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-13T11:59:53

  Same slowness, I ran :

  filer:~# gcc -shared -fPIC -ldl -o spd_readdir.so spd_readdir.c
  filer:~# LD_PRELOAD=./spd_readdir.so find /srv/vol -type d |pv -bl

  I stopped the experiment at +54min with 845k directories found (which
gives roughly the same rate of 1M directories / hour, and I know there
are more that 2M of them).

Re: ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-13T09:19:52

  I don't think I tried this specific hack, I'm having a go right now.
Is is still useful if each directory only holds a few inodes ?



  Information attached. Dumpfs said: dumpe2fs 1.42.5 (29-Jul-2012).

  Thanks for your help !
Device: /dev/vg-raid3/xyz
Blocksize: 4096 bytes
Total blocks: 78643200
Free blocks: 26099944 (33.2%)

Min. free extent: 4 KB 
Max. free extent: 557632 KB
Avg. free extent: 468 KB

HISTOGRAM OF FREE EXTENT SIZES:
Extent Size Range :  Free extents   Free Blocks  Percent
    4K...    8K-  :         52301         52301    0.20%
    8K...   16K-  :         36907         87967    0.34%
   16K...   32K-  :         36063        187694    0.72%
   32K...   64K-  :         41123        433236    1.66%
   64K...  128K-  :         26411        579943    2.22%
  128K...  256K-  :         16236        745344    2.86%
  256K...  512K-  :          6073        513906    1.97%
  512K... 1024K-  :          4378        774518    2.97%
    1M...    2M-  :           699        244092    0.94%
    2M...

Re: ext4 and extremely slow filesystem traversal

Theodore Ts'o — 2013-03-13T02:52:26

Did you sort results from readdir() by inode number?  i.e., such as
what the following LD_PRELOAD hack does?

https://git.kernel.org/cgit/fs/ext2/e2fsprogs.git/tree/contrib/spd_readdir.c?h=maint


Try running "e2fsck -fv /dev/XXX" and send me the output.

Also useful would be the output of "e2freefrag /dev/XXX" and "dumpe2fs -h"

                  - Ted

ext4 and extremely slow filesystem traversal

Vincent Caron — 2013-03-12T23:56:15

Hello list,

  I have troubles with the daily backup of a modest filesystem which
tends to take more that 10 hours. I have ext4 all over the place on ~200
servers and never ran into such a problem.

  The filesystem capacity is 300 GB (19,6M inodes) with 196 GB (9,3M
inodes) used. It's mounted 'defaults,noatime'. It sits on a hardware
RAID array thru plain LVM slices. The RAID array is a RAID5 running on
5x SATA 500G disks, with a battery-backed (RAM) cache and write-back
cache policy. To be precise, it's an Areca 1231.

  The hardware RAID array use 64kB stripes and I've configured the
filesystem with 4kB blocks and stride=16. It also has 0 reserved blocks.
In other works the fs was created with 'mkfs -t ext4 -E stride=16 -m 0
-L volname /dev/vgX/Y'. I'm attaching the mke2fs.conf for reference too.

  Everything is running with Debian Squeeze and its 2.6.32 kernel (amd64
flavour), on a 4 cores and 4 GB RAM server.

  I ran a tiobench tonight on an idle instance (I have two identicals
systems - hw, sw, data