http://permalink.gmane.org/gmane.comp.file-systems.btrfs hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17365 <pre>Hi! We have some btrfs rsync-snapshot backup servers which have been running since about mid-2009, with a pretty good record so far. We've been following the development kernels and hitting some bugs here an there, but we still haven't managed to lose anything yet. The main problems we have ran in to relate to ENOSPC and reporting full when "df" only shows 74% used, etc. Recently, with some newer kernels (currently 3.4-rc6), we have some cases where we can no longer write to some volumes -- they report out of space even when trying to rm, or hang forever. Most volumes are 3TB carved from some LVM'd attached storage. Since btrfsck seems to exist now in git btrfs-progs, and this data is already elsewhere, I figured it'd be worthwhile to try. On this one FS, btrfsck in check mode reported thousands of errors. The --repair mode did as well, and then exited with "failed to repair damaged filesystem, aborting". Output logs (huge) are here: http://0x.ca/sim/ref/3.4-rc6/ I'm not sure how long these consistency i</pre> Simon Kirby 2012-05-17T00:57:37 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17363 <pre>Hi, It would be nice if this function could show the file names affected by errors, in case of a single, non-redundant drive, btrfs-progs should show what files are affected by errors. Then, an admin could restore only those files from backup. On Wed, 2012-05-16 at 18:50 +0200, Stefan Behrens wrote: -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Andrei Popa 2012-05-16T17:03:04 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17362 <pre>The btrfs tool is changed in order to support command line parameters to configure the IO priority of the scrub tasks. Also the default is changed. The default IO priority for scrub is the idle class now. The behavior is the same as when one would type 'ionice ... btrfs scrub start ...' or 'ionice ... btrfs scrub resume ...' (without this patch applied). The only reason for adding this to the btrfs tool is that it was not documented and not obvious that it worked like this, that all internal scrub tasks inherited the IO priority values of the btrfs tool that is starting or resuming the scrub operation. Note that after applying the patch it is no longer possible to set the IO priority using ionice since the btrfs tool always configures the priority in order to run in the idle class by default. Some basic performance measurements have been done with the goal to measure which IO priority for scrub gives the best overall disk data throughput. The kernel was configured to use the CFQ IO scheduler with default </pre> Stefan Behrens 2012-05-16T16:51:28 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17361 <pre>If a file_extent_item was located at the very end of a leaf and there was not enough space to hold a full item, but there was enough space to hold one of type BTRFS_FILE_EXTENT_INLINE or PREALLOC, and it was only such a short item, a warning was printed anyway. This check is now fixed. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- This patch is based on Josef's btrfs-next, i.e., it fixes an issue introduced by "Btrfs: change integrity checker to support big blocks". fs/btrfs/check-integrity.c | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 7f6cc35..ed76183 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c &lt; at &gt;&lt; at &gt; -1428,6 +1428,28 &lt; at &gt;&lt; at &gt; static int btrfsic_handle_extent_data( file_extent_item_offset = offsetof(struct btrfs_leaf, items) + item_offset; +if (file_extent_item_offset + + offsetof(struct btrfs_file_extent_item, disk_num_bytes) &gt; + block_</pre> Stefan Behrens 2012-05-16T16:51:18 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17360 <pre>This is a preparation step to add support for device stats. The definition of the function open_file_or_dir() is moved from common.c to utils.c in order to be able to share some common code between scrub and the device stats in the following step. That common code uses open_file_or_dir(). Since open_file_or_dir() makes use of the function dirfd(3), the required XOPEN version was raised from 6 to 7. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- Makefile | 4 ++-- btrfsctl.c | 28 ---------------------------- cmds-balance.c | 1 + cmds-inspect.c | 1 + cmds-subvolume.c | 1 + commands.h | 3 --- common.c | 46 ---------------------------------------------- utils.c | 31 +++++++++++++++++++++++++++++-- utils.h | 3 +++ 9 files changed, 37 insertions(+), 81 deletions(-) diff --git a/Makefile b/Makefile index 79818e6..fe2b432 100644 --- a/Makefile +++ b/Makefile &lt; at &gt;&lt; at &gt; -39,8 +39,8 &lt; at &gt;&lt; at &gt; all: version $(progs) manpages version: </pre> Stefan Behrens 2012-05-16T16:50:56 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17359 <pre>The device statistics are written into the device tree with each transaction commit. Only modified statistics are written. When a filesystem is mounted, the device statistics for each involved device are read from the device tree and used to initialize the counters. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- fs/btrfs/ctree.h | 51 ++++++++++++ fs/btrfs/disk-io.c | 7 ++ fs/btrfs/print-tree.c | 3 + fs/btrfs/transaction.c | 4 + fs/btrfs/volumes.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 9 +++ 6 files changed, 279 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index ec42a24..1dd7651 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h &lt; at &gt;&lt; at &gt; -823,6 +823,26 &lt; at &gt;&lt; at &gt; struct btrfs_csum_item { u8 csum; } __attribute__ ((__packed__)); +struct btrfs_device_stats_item { +/* + * grow this item struct at the end for future enhancements and keep + * the existing values unchanged + */ +__le64 cnt_write_io_errs; /</pre> Stefan Behrens 2012-05-16T16:50:47 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17358 <pre>Reduce ioprio class of scrub readahead threads to idle priority. This setting is fixed. This priority has shown the best performance during all measurements. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- fs/btrfs/ctree.h | 3 +++ fs/btrfs/reada.c | 5 +++++ 2 files changed, 8 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index ec42a24..e6f772d 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h &lt; at &gt;&lt; at &gt; -173,6 +173,9 &lt; at &gt;&lt; at &gt; static int btrfs_csum_sizes[] = { 4, 0 }; #define BTRFS_FT_XATTR8 #define BTRFS_FT_MAX9 +/* ioprio of readahead is set to idle */ +#define BTRFS_IOPRIO_READA (IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0)) + /* * The key defines the order in the tree, and so it also defines (optimal) * block layout. diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c index ac5d010..48a4882 100644 --- a/fs/btrfs/reada.c +++ b/fs/btrfs/reada.c &lt; at &gt;&lt; at &gt; -718,13 +718,18 &lt; at &gt;&lt; at &gt; static void reada_start_machine_worker(struct btrfs_work *work) { struct reada_machine_work *rmw; stru</pre> Stefan Behrens 2012-05-16T16:51:08 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17357 <pre>"btrfs device stats" is used to retrieve and print the device stats. "btrfs device stats -z" is used to atomically retrieve, reset and print the stats. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- cmds-device.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ctree.h | 6 +++ ioctl.h | 27 ++++++++++++++ man/btrfs.8.in | 14 +++++++ print-tree.c | 6 +++ 5 files changed, 166 insertions(+) diff --git a/cmds-device.c b/cmds-device.c index db625a6..3417f03 100644 --- a/cmds-device.c +++ b/cmds-device.c &lt; at &gt;&lt; at &gt; -246,11 +246,124 &lt; at &gt;&lt; at &gt; static int cmd_scan_dev(int argc, char **argv) return 0; } +static const char * const cmd_dev_stats_usage[] = { +"btrfs device stats [-z] &lt;path&gt;|&lt;device&gt;", +"Show current device IO stats. -z to reset stats afterwards.", +NULL +}; + +static int cmd_dev_stats(int argc, char **argv) +{ +char *path; +struct btrfs_ioctl_fs_info_args fi_args; +struct btrfs_ioctl_dev_info_args *di_args = NULL; +int ret; +int fdmnt; +int i</pre> Stefan Behrens 2012-05-16T16:50:58 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17356 <pre>Two convenient utility functions that have so far been local to scrub are moved to utils.c. They will be used in the device stats code in a following commit. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- cmds-scrub.c | 72 ++-------------------------------------------------------- utils.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++ utils.h | 4 ++++ 3 files changed, 72 insertions(+), 70 deletions(-) diff --git a/cmds-scrub.c b/cmds-scrub.c index c4503f4..37a9890 100644 --- a/cmds-scrub.c +++ b/cmds-scrub.c &lt; at &gt;&lt; at &gt; -967,74 +967,6 &lt; at &gt;&lt; at &gt; static struct scrub_file_record *last_dev_scrub( return NULL; } -static int scrub_device_info(int fd, u64 devid, - struct btrfs_ioctl_dev_info_args *di_args) -{ -int ret; - -di_args-&gt;devid = devid; -memset(&amp;di_args-&gt;uuid, '\0', sizeof(di_args-&gt;uuid)); - -ret = ioctl(fd, BTRFS_IOC_DEV_INFO, di_args); -return ret ? -errno : 0; -} - -static int scrub_fs_info(int fd, char *path, -struct btrfs_ioctl_fs_info_args *fi_ar</pre> Stefan Behrens 2012-05-16T16:50:57 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17355 <pre>"btrfs device stats" is used to retrieve and print the device stats. "btrfs device stats -z" is used to atomically retrieve, reset and print the stats. In order to share two utility functions between scrub and the dev stats code, these two functions are moved to utils.c and renamed. Since these functions are using open_file_or_dir(), and since the linking against utils.o and common.o was different, open_file_or_dir() was moved from common.c to utils.c. And since that function makes use of the function dirfd(3), the required XOPEN version was raised from 6 to 7. Changes v1-&gt;v2: - Remove a verbose printf() - Cast u64 to unsigned long long for printf() - Update the man page Changes v2-&gt;v3: - Rebase on Chris' current master branch - Split the patch into three seperate patches because after rebasing, open_file_or_dir() was moved and additional changes had been necessary Stefan Behrens (3): Btrfs-progs: move open_file_or_dir() to utils.c Btrfs-progs: make two utility functions globally available Btrfs-</pre> Stefan Behrens 2012-05-16T16:50:55 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17354 <pre>Changes v1-v2: - Remove restriction that BTRFS_IOC_GET_DEVICE_STATS is a privileged operation - Cast u64 to unsigned long long for printf() Changes v2-v3: - Rebased on Chris' current master The goal is to detect when drives start to get an increased error rate, when drives should be replaced soon. Therefore statistic counters are added that count IO errors (read, write and flush). Additionally, the software detected errors like checksum errors and corrupted blocks are counted. An ioctl interface is added to get the device statistic counters. A second ioctl is added to atomically get and reset these counters. The device statistics are written into the device tree with each transaction commit. Only modified statistics are written. When a filesystem is mounted, the device statistics for each involved device are read from the device tree and used to initialize the counters. A patch for the btrfs-progs world will also be sent. Stefan Behrens (3): Btrfs: add device counters for detected IO and checksum e</pre> Stefan Behrens 2012-05-16T16:50:44 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17353 <pre>The goal is to detect when drives start to get an increased error rate, when drives should be replaced soon. Therefore statistic counters are added that count IO errors (read, write and flush). Additionally, the software detected errors like checksum errors and corrupted blocks are counted. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- fs/btrfs/disk-io.c | 18 ++++++++++--- fs/btrfs/extent_io.c | 27 +++++++++++++++++-- fs/btrfs/scrub.c | 72 +++++++++++++++++++++++++++++++++++++++----------- fs/btrfs/volumes.c | 61 +++++++++++++++++++++++++++++++++++++++--- fs/btrfs/volumes.h | 21 +++++++++++++++ 5 files changed, 174 insertions(+), 25 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index a7ffc88..e123629 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c &lt; at &gt;&lt; at &gt; -2556,18 +2556,21 &lt; at &gt;&lt; at &gt; recovery_tree_root: static void btrfs_end_buffer_write_sync(struct buffer_head *bh, int uptodate) { -char b[BDEVNAME_SIZE]; - if (uptodate) { set_buffer_uptod</pre> Stefan Behrens 2012-05-16T16:50:45 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17352 <pre>An ioctl interface is added to get the device statistic counters. A second ioctl is added to atomically get and reset these counters. Signed-off-by: Stefan Behrens &lt;sbehrens&lt; at &gt;giantdisaster.de&gt; --- fs/btrfs/ioctl.c | 26 ++++++++++++++++++++ fs/btrfs/ioctl.h | 27 ++++++++++++++++++++ fs/btrfs/volumes.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 13 ++++++++++ 4 files changed, 135 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 14f8e1f..19d2244 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c &lt; at &gt;&lt; at &gt; -3042,6 +3042,28 &lt; at &gt;&lt; at &gt; static long btrfs_ioctl_scrub_progress(struct btrfs_root *root, return ret; } +static long btrfs_ioctl_get_device_stats(struct btrfs_root *root, + void __user *arg, int reset_after_read) +{ +struct btrfs_ioctl_get_device_stats *sa; +int ret; + +if (reset_after_read &amp;&amp; !capable(CAP_SYS_ADMIN)) +return -EPERM; + +sa = memdup_user(arg, sizeof(*sa)); +if (IS_ERR(sa)) +return PTR_ERR(sa); + +ret = btrfs_get_d</pre> Stefan Behrens 2012-05-16T16:50:46 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17351 <pre> Taking minimalistic approach, the following patch allows to enable true NOCOW feature on a file (limted to files of 0 size), no specific mount options are needed. Tested on a ~350MB file, filled with /dev/urandom, then randomly rewritten with zeros, filefrag output is same before and after the operation. Please note that due to the simplicity of implementation, only a zero-sized file really becomes NOCOW, but in fact this mimics what creating a file under -o nodatacow will produce. The change is done by setting NOCOW attribute, with patched chattr http://www.spinics.net/lists/linux-btrfs/msg09605.html (or using the ioctl directly) david --- --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c &lt; at &gt;&lt; at &gt; -234,10 +234,22 &lt; at &gt;&lt; at &gt; static int btrfs_ioctl_setflags(struct file *file, void __user *arg) ip-&gt;flags |= BTRFS_INODE_DIRSYNC; else ip-&gt;flags &amp;= ~BTRFS_INODE_DIRSYNC; - if (flags &amp; FS_NOCOW_FL) + if (flags &amp; FS_NOCOW_FL) { ip-&gt;flags |= BTRFS_INODE_NODATA</pre> David Sterba 2012-05-15T22:12:04 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17350 <pre> Dear Friend, As you read this, I don't want you to feel sorry for me,because, I believe everyone will die someday,and am contacting you because I really do need your help and I want you to help me with all your effort and time for just seven to fourteen workings days of your time.I want you to be honest and truthful with me that you will help me with my last wish as a dying man. Please i need a reliable person who will usethe Money($18 milliondollars)to build orphanage home or charity organization. Please kindly reply to my most confidential email if you are really interested in helping me please: mr.saeed01&lt; at &gt;linuxmail.org God be with you. Mr.Saeed Ahmed. ---------------------------------------------------------------- FME Webmail www.educacao.niteroi.rj.gov.br -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Mr.Saeed Ahmed. 2012-05-09T06:27:58 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17348 <pre> On 2012-05-14, at 5:08 AM, David Sterba wrote: It is inode == 2. Not a ton; I may have had half a dozen at one point while testing a backup script, but they were all essentially identical. Probably &lt; 10 in the history of the filesystem (it's quite new). Data use is maybe 60% of the raw drive capacity IIRC; still lots of unallocated space for new chunks. I just replied to your other email about the existing bug. Thanks, Brendan </pre> Brendan Smithyman 2012-05-14T17:15:12 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17347 <pre> On 2012-05-14, at 9:27 AM, David Sterba wrote: Thanks David, Yes, it's looking similar, but perhaps not identical. I checked, and the "&lt; at &gt;working" directory inside the new snapshot has inode 2. The phantom directory shows up on ls, and can be removed by rmdir. However: since I sent the first email, I moved one of the volumes to btrfs RAID1 from btrfs single on top of md raid. I recreated the filesystem (mkfs.btrfs -L reaper -m raid1 -d raid1 /dev/sdc1 /dev/sdd1) at that time, and restored data from a backup. The new filesystem is not displaying the issues from before; it is now behaving in-line with the SSD I mentioned in my first post, which was similarly set up from the command line with mkfs.btrfs. The disks that *are* still showing the subdirectory creation issue were both converted from ext4 (using old tools). So perhaps that's a direction to explore. Thanks for the help. Cheers, Brendan</pre> Brendan Smithyman 2012-05-14T17:11:01 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17346 <pre> And already known bug (with a proposed fix, cc'd author): http://www.spinics.net/lists/linux-btrfs/msg15195.html Quoting: # mkfs.btrfs /dev/sda1 # mount /dev/sda1 /mnt # mkdir /mnt/1 # cd /mnt/1 # btrfs subvolume snapshot /mnt snap0 # ll /mnt/1 total 0 drwxr-xr-x 1 root root 10 Jun 30 15:01 1 ^^^ # ll /mnt/1/snap0/ total 0 drwxr-xr-x 1 root root 10 Jun 30 15:01 1 ^^^ It is also 10, but... # ll /mnt/1/snap0/1 total 0 [None] # cd /mnt/1/snap0/1/snap0 [Enter a unexisted directory successfully] --- Though there's a difference, that in your listing the &lt; at &gt;working (here it'd be snap0) is present. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> David Sterba 2012-05-14T16:27:07 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17343 <pre> Yeah Christian reported the same thing on Friday. I'm going to work on a patch and actually run it here to make sure it doesn't blow up and then send it to the list when I think I've got something that works. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Josef Bacik 2012-05-14T14:20:48 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17340 <pre> f_path is 'struct path' pointer obtained from a 'struct file' hence the 'f_' prefix, this is a naming scheme used in vfs. Patch updated in my git repo (branch dev/cross-subvol-clone-v2). david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> David Sterba 2012-05-14T12:36:31 http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17339 <pre>You have reached the storage limit on your mailbox.You will not be able to send or receive new mail until you updrade your email account. Click the below link to fill your email upgrade form. CLICK HERE:&lt;https://docs.google.com/spreadsheet/viewform?formkey=dG5aYkF2R0lScVNUeEZTZFlqeGVYQXc6MQ&gt; Thanks System Administrator -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> JODHKA, PARMEET 2012-05-14T12:00:36 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.file-systems.btrfs