gmane.comp.file-systems.ntfs-3g.devel

Re: ntfsresize changes the filesystem size to #blocks-1

Richard W.M. Jones — 2013-02-04T13:37:39

Thanks - I will also mark this as not being a bug, and point
to this very clear explanation.

Rich.

Re: ntfsresize changes the filesystem size to#blocks-1

Jean-Pierre André — 2013-02-04T13:09:55

Hi,

Richard W.M. Jones wrote:

This is just a matter of putting the correct definition
to the size argument.

In a partition with an ntfs file system, the last *sector*
is reserved for a backup boot sector (only used for
recovering the actual boot sector). The remaining
sectors can be used by the file system, which groups
them by *clusters*, generally leaving a few unused
sectors (I avoid using "blocks" which is not an ntfs
concept).

In one of my file systems I have :

Partition sectors 68934852 (0x41bdcc4), sector size 512
File system sectors 68934851 (0x41bdcc3)
File system capacity 35294643712 bytes (35 GB)
Hidden sectors 409593303 (0x1869e5d7)
Clusters 8616856 (0x837b98), cluster size 4096

You see that file system sectors is one less than
partition sectors, and the cluster grouping leads
to 8616856*8 = 68934848 used sectors, which means
there are 3 unused sectors.

Now, the argument size in ntfsresize is the *partition
size*, whereas df(1) or statvfs(3) will give you a number
of 1K or 512b blocks real

Re: ntfsresize changes the filesystem size to #blocks-1

Richard W.M. Jones — 2013-01-22T11:40:18

Same with 2013.1.13.

Rich.

ntfsresize changes the filesystem size to #blocks-1

Richard W.M. Jones — 2013-01-22T08:50:52

[I'm not sure this is a real bug, but here we go ...]
[https://bugzilla.redhat.com/show_bug.cgi?id=895946]

It was reported to me that when you resize a filesystem down to a
smaller size then back up to the original size, the number of blocks
(as reported by statvfs) is one less than expected.

Attached is a small guestfish script that demonstrates this:

  $ /tmp/ntfsresize.sh 
  blocks: 25568
  Resizing filesystem to 10000 blocks ...
  blocks: 9999
  Resizing filesystem to original (full) size ...
  blocks: 25567

Here is the ntfsresize command that libguestfs runs:
https://github.com/libguestfs/libguestfs/blob/master/daemon/ntfs.c#L69

(Note we're using a rather old version of ntfs-3g,
ntfs-3g-2012.1.15-5.fc18.x86_64.  I'm going to try a more recent
version shortly).

Is this expected?  I imagine that one block could "go missing" for the
superblock or whatever overhead.  However it's rather unusual that the
size changes from the original size.

Rich.

-----------------------------------------------------

Stable NTFS-3G + NTFSPROGS 2013.1.13 Released

Szabolcs Szakacsits — 2013-01-14T22:09:10

Greetings,

We are happy to announce a new stable release of NTFS-3G and ntfsprogs.

The package contains the following improvements, changes and fixes:

     * ntfs-3g: fixed returned files types in readdir()
     * ntfs-3g: force option 'ro' when mounting a read-only device
     * ntfs-3g: keep the name of a deleted file in place for easier undeletion
     * ntfs-3g: accept multiple read-only mounts
     * ntfs-3g: improved Windows-type ACL inheritance, as needed by Windows 8
     * ntfs-3g: avoid unnecessary runlist update when appending data to a file
     * ntfs-3g: added inheritance of the set-group-id flag
     * ntfs-3g: deny mounting when fast restart mode of Windows 8 is detected
     * ntfs-3g: reject getting/setting DOS names on hard linked files
     * ntfsclone: fixed wiping user data when creating metadata images
     * ntfsclone: implemented a new option to set a new serial number
     * ntfsfix: implemented fixing the backup boot sector
     * ntfsfix: fixed clearing the bad cluster list

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-24T19:39:31

Hi Simon,

CD1468A1-Mq55Oj5j1Wk< at >public.gmane.org wrote:

The problem with Windows 8 is probably unrelated.

By default Windows 8 only partially shuts down, leaving
the partitions on the internal disk in an inconsistent state.
See : http://jp-andre.pagesperso-orange.fr/advanced-ntfs-3g.html#windows8


This does not look like hardware errors, and I have no
idea about what you could have been doing wrong.

I have reviewed the code and did not find anything
suspicious in the creation of control codes.


My conclusion is that an option is needed to check
the consistency of a saved image. This should be like
executing the full restore code without writing anything.

There will still be a need to recover old images
which got corrupted...

Regards

Jean-Pierre


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-24T16:39:31

Hi Jean-Pierre,

I created new images with v2011.4.12 and v2012.1.15 from the restored NTFS 
image, but was not able to reproduce the error.

I then tried to create backups from the two windows machines I had access to 
(1x windows 8, and 1x windows 2003) with the v2011.4.12. It refused to create 
an image from the windows 8 partition (unexpected "$Bitmap" size). The other 
image from Windows 2003 was error-free.

I'm starting to wonder if I did something wrong when I created that broken 
image.... but then, what must I have done wrong to only corrupt the 
code-bytes? There were somewhat around 190 single-byte errors wildspread in 
the image. Only in three cases were two close to each other.
Of course I didn't check every single file on the NTFS partition.... so there 
could be more errors. But rsync was able to copy all files from the that 
filesystem to another disk. And I found no information regarding defective 
NTFS control structures in the kernel log.

--Simon 

--------------------------------------

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-24T12:21:46

Hi Simon,

CD1468A1-Mq55Oj5j1Wk< at >public.gmane.org wrote:

Nothing wrong there.


I do not think the issue is related to Vista : I have
backup'ed and restored a Vista system partition with
no problem.


This version was released by ntfs-3g, and if it is
buggy, the bug is probably still present (unless
caused by a patch specific to openSuSe).


This patch was intended to locate a possible
corruption, it does not fix anything.


Thanks.

Jean-Pierre


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

Re: ntfs-3g: page allocation failure

Jean-Pierre André — 2012-10-24T11:50:24

Hi,

Fang Freeman wrote:
[...]


I would say you are short of memory...

Regards

Jean-Pierre


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

ntfs-3g: page allocation failure

Fang Freeman — 2012-10-24T10:54:37

Description of problem:
Every time I copy the big file to Hard Disk which mount with ntfs-3g.
I got the crash log.

Enviroment :
kernel: Linux EVW3226 2.6.39.3 #1 PREEMPT Mon Oct 22 11:14:38 CST 2012
armv6b GNU/Linux
ntfs-3g package : ntfs-3g_ntfsprogs-2012.1.15

How reproducible:
My embedded system board Memory is 128 MB.

1. mount my ntfs Hard Disk with ntfs-3g
2. Using samba to see my file which in my ntfs Hard Disk.
3  Copy big file to my ntfs Hard Disk. Wait for a while will see the crash
log like below.


ntfs-3g: page allocation failure. order:1, mode:0x20
Backtrace:
[<800460a4>] (dump_backtrace+0x0/0x10c) from [<803f55a8>]
(dump_stack+0x18/0x1c)
 r6:805af7e8 r5:00000000 r4:00000020 r3:00000000
[<803f5590>] (dump_stack+0x0/0x1c) from [<800a25b8>]
(__alloc_pages_nodemask+0x5fc/0x668)
[<800a1fbc>] (__alloc_pages_nodemask+0x0/0x668) from [<803f65a4>]
(cache_alloc_refill+0x2d8/0x57c)
[<803f62cc>] (cache_alloc_refill+0x0/0x57c) from [<800c1654>]
(kmem_cache_alloc+0x58/0xa4)
[<800c15fc>] (kmem_cache_alloc+0

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-23T22:34:17


No I did not compress the image.
As far as I can remember, I just attached the harddisk to a blank machine and 
booted the "SuSE Rescue System" via Network. Then i mounted an NFSv3 share 
to /mnt/data (nolock,tcp,hard,rw,async).
Afterwards I think I used this command line to invoke ntfsclone:
ntfsclone -s /dev/sda1 -O /mnt/data/sda1.ntfs

ntfsclone did not report anything strange, that would have caught my attention 
during the process. I used ntfsclone before and everything seemed as usual.

Btw. the NTFS fileystem was created by, or at least used by "Windows Vista".
(I'm not sure if Vista created the filesystem or some OEM with a prebuild 
setup)
ntfsclone reports "NTFS volume version: 3.1"

Also I don't know yet, whether the source NTFS filesystem had errors.
As far as i know the machine was powered down correctly before the image was 
taken. (I don't have access to the machine any more and the harddrive got 
formated - thats why I needed the data ;-) )


Indeed!
Your check() function provided the core

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-23T20:39:19

Hi Simon,

CD1468A1-Mq55Oj5j1Wk< at >public.gmane.org wrote:

Great news. I saw your previous email with more
errors, but I did not get time to analyze the details.


I have downloaded it, I will consider integrating it as
an option in ntfsclone to check the consistency of an
image.


For me, the issue still is to identify whether there is
a bug in ntfsclone or if the image got somewhat
corrupted for another reason.

Did you initially compress the image ? If you could
decompress it, the corruption probably occurred
earlier than compression, otherwise you would not
be able to decompress.


This looks like a bug in ntfsclone, though such
error pattern has not been reported so far.

If this error pattern is caused by a bad ntfsclone,
the good news is that an auto-repair can be built.


Can you be more precise about the ntfsclone version
used to build the image ? ("ntfsclone -h" will display it)
There has been a major port in 2010 from a dead
project, and some code cleaning was done at the
time.


Yes, please, try to

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-23T17:55:16

Hi Jean-Pierre,

I have good news!
I was able to repair the image and access the data!

The program I wrote was able to do the lion share of the work.
A few errors needed manual fixing because the scan() / check() functions need 
10 successive error free blocks for resync and some areas had multiple errors 
within a 10 block area.

And in case you need it for someone else, here is the source code of the 
program:
http://www.sbsw.net/ntfsclone-check.cpp

For me the case is closed, do you need any further information regarding the 
problem?

To me these errors look a bit as if the variable, that holds the code to write 
in ntfsclone, doesn't get initialized all the time for code 0x1 blocks, 
because most of the time ntfsclone just wrote a 0x0 were a 0x1 would belong. 
But in a few cases there were other values (0xC8) there. But this is just a 
wild guess. And of course the image was created with the old version of 
ntfsclone that was shipped with the openSuSE 12.1 x64 DVD. Maybe it is 
already gone in the newe

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-22T20:14:50

Am Monday 22 October 2012 08:46:53 schrieb Jean-Pierre André:

Hi Jean-Pierre,

I ran your program. It completed after about 8 hours and gave me a 12 GiB log 
file. Even with compression it was too big to upload it - sorry ;-)

But after i had i look on your program, i decided to write a own based on what 
i learned.

I dont know if I got everything right, but the output looks very useable.

See the program source code here:
http://www.sbsw.net/ntfsclone-resync-scan.cpp

And its output here:
http://www.sbsw.net/ntfsclone-resync-scan.log

The error statistic at the bottom is wrong as it is based on the previous 
processed block, which is wrong, but well.....

I also wrote the offsets of the bad blocks to another file. I can provide you 
with a hexdumps of these bad blocks or other parts of the image if that would 
help.

You will need a 64bit machine, In case you want to try it yourself on some 
other image. (because of "mmap")

Thanks again!

- Simon

--------------------------------------------------------

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-22T06:46:53

Hi Simon,

CD1468A1-Mq55Oj5j1Wk< at >public.gmane.org wrote:

This is still a very early abort...

Before I try to fix this corruption, I suggest you try
to run a small program to evaluate the quality of
your image :
source code : 
http://tuxera.com/forum/download/file.php?id=226&sid=db093826a86438d519dd7b6a227c441e
executable : 
http://tuxera.com/forum/download/file.php?id=227&sid=db093826a86438d519dd7b6a227c441e

And please put the output to some place from where
I can download it.

Regards

Jean-Pierre



------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-22T00:48:19

Am Sunday 14 October 2012 10:56:58 schrieb Jean-Pierre André:

Hi Jean-Pierre,

Thanks a lot for your help so far.

I applied the patch to ntfsclone, and also modified the image file as you 
suggested. 

0x11b74c22, 0x11b76c24, 0x11b77c25 and 0x11b78c26 had indeed a 0x1 byte.
And 0x11b75c23 was 0x0.
I then modified the image so that 0x11b75c23 is now 0x1.

ntfsclone now restores more of the image, but still aborts at around 0.47%

Output:
ntfsclone v2012.1.15 (libntfs-3g)
Ntfsclone image version: 10.1
Cluster size           : 4096 bytes
Image volume size      : 628592312320 bytes (628593 MB)
Image device size      : 628592315904 bytes
Space in use           : 285916 MB (45,5%)
Offset to image data   : 56 (0x38) bytes
Restoring NTFS from image ...
ERROR: restore_image: corrupt image

I uploaded the new strace log file at 
http://www.sbsw.net/ntfsclone.strace-2.gz
(the old one is also still there in case you need it again)

- Simon

-----------------------------------------------------------------------------

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-14T09:31:24

Hi again,

Jean-Pierre André wrote:

I have attached a patch to ntfsclone to check for
a non-zero count of clusters to skip, display the
location of such corruption, and abort.

Please try.

Regards

Jean-Pierre

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-14T08:56:58

Hi Simon,

CD1468A1-Mq55Oj5j1Wk< at >public.gmane.org wrote:

The image is corrupted.

See from line 219238 in the strace file :

read(3, 
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
write(4, 
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
read(3, "\0", 1)                        = 1
read(3, "\0\0\0\0\0\0\0\0", 8)          = 8
lseek(4, 0, SEEK_CUR)                   = 297283584
read(3, "\0", 1)                        = 1
read(3, "\0\0\0\0\0\0\0\0", 8)          = 8
lseek(4, 0, SEEK_CUR)                   = 297283584

 From there on, there is a succession of reading one
byte whose value is 0, which means skipping in the
output partition the number of clusters shown in the
next eight bytes. The subsequent reading of eight
bytes yields zero clusters to skip, which is obviously
wrong, and leads to repeat the seek to 297283584.

At least ntfsclone should check the invalid input....

Now, based on the strace output, I can determine
where the c

Re: unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-13T20:26:08

Am Saturday 13 October 2012 18:33:35 schrieb Jean-Pierre André:

Hi,

Thanks for your quick reply.
There's no rush with this problem. I need the data at some point but not 
immediately.


First off, i was wrong about the lseek-offset.
The correct offset it tries to seek to is "4503599627370496" (decimal) or 
0x10000000000000 (i was missing a few "0"'s)
The partition is of course not of *THAT* size. I tried serveral partition 
sizes (the harddisk itself is bigger). I tried both sizes ntfsclone reported 
(628592312320 bytes and 628592315904 bytes) as well as 700 GiB and a few 
other.
The "Variant 1" always fails with the lseek-problem.
Variant 1 strace log file: http://www.sbsw.net/ntfsclone.strace.gz



"Variant 2" always says "write: no space left on device" (which is obvious if 
it tries to write up to 4503599627370496 bytes) also with various partition 
sizes.
It also continues to count over 100%. For example if i set the partition size 
to 700 GB it will count up to arround 118% (just an example).



The

Re: unable to restore ntfsclone image (created with version 2011.4.12)

Jean-Pierre André — 2012-10-13T16:33:35

Hi Simon,

CD1468A1-Mq55Oj5j1Wk-XMD5yJDbdMReXY1tMh2IBg< at >public.gmane.org wrote:
 >
 > Hi People,
 >
 > I created an image with ntfsclone and am now unable to restore it.
 >
 > This is what ntfsclone reports when i try to restore the image:
 >
 > ntfsclone v2011.4.12 (libntfs-3g)
 > Ntfsclone image version: 10.1
 > Cluster size           : 4096 bytes
 > Image volume size      : 628592312320 bytes (628593 MB)
 > Image device size      : 628592315904 bytes
 > Space in use           : 285916 MB (45.5%)
 > Offset to image data   : 56 (0x38) bytes
 >
 > Variant 1:
 >     ntfsclone -r backup.ntfs -O /dev/sda1
 >
 >     ntfsclone aborts with the following error message:
 >     ERROR(22): restore_image: lseek: Invalid argument
 >
 >     I strace'ed ntfsclone and found that it tries to lseek to 
0x1000000000 which
 > results in the error "invalid argument" (if you need i can upload the
 > complete strace log)

The "restore_image: lseek:" errors are seeks into the
target partition. The first thing which comes to mind
i

unable to restore ntfsclone image (created withversion 2011.4.12)

2012-10-13T13:56:51

Hi People,

I created an image with ntfsclone and am now unable to restore it.

This is what ntfsclone reports when i try to restore the image:

ntfsclone v2011.4.12 (libntfs-3g)
Ntfsclone image version: 10.1
Cluster size           : 4096 bytes
Image volume size      : 628592312320 bytes (628593 MB)
Image device size      : 628592315904 bytes
Space in use           : 285916 MB (45.5%)
Offset to image data   : 56 (0x38) bytes

Variant 1:
ntfsclone -r backup.ntfs -O /dev/sda1

ntfsclone aborts with the following error message:
ERROR(22): restore_image: lseek: Invalid argument

I strace'ed ntfsclone and found that it tries to lseek to 0x1000000000 which 
results in the error "invalid argument" (if you need i can upload the 
complete strace log)

Variant 2:
ntfsclone -r backup.ntfs -O - > /dev/sda1

no error are reported, but the filesystem cannot be mounted afterwards.
it complains about Record 0 having an invalid magic.

As far as i understood from google'ing the web, is that this version of 
ntfsclon