gmane.os.freebsd.devel.hackers

Re: find -delete broken, or just used improperly?

Kurt Lidl — 2013-05-21T18:04:55

Using "-mindepth 1" does indeed preserve the target directory.
Without it, the target directory is removed if all the children
files/directories are empty.

I've just finished a complete build of a recent stable/9:

make buildworld && make buildkernel && cd release && make release

With WITHOUT_SHAREDOCS= set in my src.conf, and Jilles' patch to
find, and the following patch to Makefile.inc1, it now builds a
release properly.

bhyve9# hg diff Makefile.inc1
diff -r ca8ef48b4ba6 Makefile.inc1
--- a/Makefile.inc1Thu May 16 10:21:04 2013 -0400
+++ b/Makefile.inc1Tue May 21 13:58:13 2013 -0400
< at >< at > -754,7 +754,7 < at >< at >
      ${IMAKEENV} rm -rf ${INSTALLTMP}
  .if make(distributeworld)
  .for dist in ${EXTRA_DISTRIBUTIONS}
-find ${DESTDIR}/${DISTDIR}/${dist} -empty -delete
+find ${DESTDIR}/${DISTDIR}/${dist} -mindepth 1 -empty -delete
  .endfor
  .if defined(NO_ROOT)
  .for dist in base ${EXTRA_DISTRIBUTIONS}

If one doesn't have the "-mindepth 1" in the find command, it will
remove the usr/share/doc directory ent

Re: find -delete broken, or just used improperly?

John Baldwin — 2013-05-21T15:06:39

I'm curious, how would you instruct a patched find to avoid deleteing the
/tmp/foo directory (e.g. if you wanted this to be a job that pruned empty
dirs from /tmp/foo but never pruned the directory itself).  Would -mindepth 1
do it?  (Just asking.  I have also found this message annoying but most of
the jobs I have seen it on probably don't want to delete the root path,
just descendants.)

Re: find -delete broken, or just used improperly?

Jilles Tjoelker — 2013-05-20T21:47:31






The "relative path" refers to a pathname that contains a slash other
than at the beginning or end and may therefore refer to somewhere else
if a directory is concurrently replaced by a symlink.

When -L is not specified and "." can be opened, the fts(3) code
underlying find(1) is careful to avoid following symlinks or being
dropped in different locations by moving the directory fts is currently
traversing. If a problematic concurrent modification is detected, fts
will not enter the directory or abort. Files found in the search are
returned via the current working directory and a pathname not containing
a slash.

For paranoia, find(1) verifies this when -delete is used. However, it is
too paranoid about the root of the traversal. It is already assumed that
the initial pathname does not refer to directories or symlinks that
might be replaced by untrusted users; otherwise, the whole traversal
would be unsafe. Therefore, it is not necessary to do the check for
fts_level == FTS_ROOTLEVEL.

The below patch a

Re: Adding a FOREACH_CONTINUE() variant to queue(3)

John Baldwin — 2013-05-20T19:39:57

http://people.freebsd.org/~lstewart/patches/misc/queue_foreach_from_10.x.r250136.patch

Looks ok to me.  I agree with phk< at > and prefer the _FROM name.

find -delete broken, or just used improperly?

Kurt Lidl — 2013-05-20T19:23:16

OK, maybe I'm missing something obvious, but...

find(1) says:

     -delete
             Delete found files and/or directories.  Always returns true.
             This executes from the current working directory as find recurses
             down the tree.  It will not attempt to delete a filename with a
             ``/'' character in its pathname relative to ``.'' for security
             reasons.  Depth-first traversal processing is implied by this
             option.  Following symlinks is incompatible with this option.

However, it fails even when the path is absolute:

bhyve9# mkdir /tmp/foo
bhyve9# find /tmp/foo -empty -delete
find: -delete: /tmp/foo: relative path potentially not safe

Shouldn't this work?

I ran into this during a build of stable/9 with WITHOUT_SHAREDOCS
set, which ultimately triggers this bit of /usr/src/Makefile.inc1:

.for dist in ${EXTRA_DISTRIBUTIONS}
        find ${DESTDIR}/${DISTDIR}/${dist} -empty -delete
.endfor

The actual observed failure is this:

===> etc/sendmail (d

Re: blogbench and write-open serialization

2013-05-20T17:49:13

On Sat, May 18, 2013 at 5:59 AM, Florent Peterschmitt
<florent< at >peterschmitt.fr> wrote:

I largely reproduced your results, but saw even worse scaling.
* Linux was much faster, for all thread counts and on all file systems
* Linux's performance degraded by about 25% at high thread counts.
FreeBSD's penalty was file-system dependent, but usually higher
* The filesystem and hard disk mattered little on linux (tested tmpfs
and ext4).  The hard disk mattered little on FreeBSD, but the file
system was very significant.
* On FreeBSD, tmpfs was the fasted for a single thread.  For nine
threads, tmpfs was fastest for writes (about 33% faster than UFS or
ZFS), but substantially slower than ZFS for reads.
* For O_RDONLY, ZFS scales much better than UFS as the number of
threads increases, but at low thread counts UFS was faster.

I was testing on a dual-socket, 6 core/socket Xeon E5645 < at > 2.4GHz
machine.  For Linux, I used Debian 6.0.6 with the 2.6.32-5 kernel.
For FreeBSD, I was using a custom kernel based on stable/9.

Re: Managing userland data pointers in kqueue/kevent

Jilles Tjoelker — 2013-05-19T16:17:38











An important detail is missing: how do you avoid using up all kernel
memory on knotes if someone keeps adding new file descriptors with
EV_ADD | EV_DROPWATCH and closing the file descriptors again without
ever draining the kqueue?

This problem did not use to exist for file descriptor events before: the
number of such knotes was limited to the number of open file
descriptors. However, it does already exist for most of the other event
types. For example, pwait -v will return the exit status even if it was
suspended (^Z) while the process terminated and the parent reaped the
zombie. For EVFILT_TIMER, the worst effect is a denial of service of
EVFILT_TIMER on all other processes in the system. EVFILT_USER does not
appear to check anything and appears to allow arbitrary kernel memory
consumption.

The EVFILT_TIMER needs to keep its global limit and EVFILT_USER needs
something similar.

For the rest, call an event that is no longer associated to a kernel
object (e.g. EVFILT_READ whose file descriptor i

Re: Managing userland data pointers in kqueue/kevent

Dirk Engling — 2013-05-18T18:32:05


I earlier pointed out other things that prevent me from using kqueue as
a proper storage for user land pointers:

http://lists.freebsd.org/pipermail/freebsd-hackers/2013-March/042181.html

To sum it up, once I pass a pointer via udata to the kqueue system,
kqueue becomes the owner and the expected behaviour is

  (a) never silently swallow the pointer
  (b) provide an interface to retrieve the pointer anytime

Besides the way you pointed out to violate (a), there is another way,
re-adding an existing event. So I propose a flag EV_EXCL that will fail
adding an existing event with EEXIST in data.

As an alternative or in addition to that, re-adding an existing event
without the EV_EXCL flag will cause the old event to be returned with
the EV_DROPPED mechanism proposed.

This can also be used to fulfill property (b). kqueue is an efficient
store for the per-event-data. In an event base application, I usually
allocate resources per new session, pass the metadata via udata to
kevent and will see the udata point

Re: blogbench and write-open serialization

Florent Peterschmitt — 2013-05-18T11:59:15

Le 18/05/2013 06:04, Ivan Voras a écrit :

Hi,

I'm no able to say anything about that (because I've no idea of how does 
Linux or FreeBSD works ), but could it be a problem from filesystem ?

Everytime I had UFS I found the entire system very slow when doing some 
I/O (many little freezes), and with ZFS it's globally much better.
_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

blogbench and write-open serialization

Ivan Voras — 2013-05-18T04:04:05

During the BSDCan & DevSummit I got interested in finding out why
blogbench is so slow on FreeBSD. After talking to jhb, it looked like
one of the reasons might be that opening files with O_RDWR or O_WRONLY
(anything opening the file for writing) is serialized.

To check this, I've written a small test program, which I've run on
CentOS 6.3 and FreeBSD 10-HEAD on the same hardware. Here are the results:

https://wiki.freebsd.org/Benchmarking/OpenCloseBenchmark

Conclusions:

* Linux opens and closes files much faster than FreeBSD
* Linux does not serialize write-open operations, while FreeBSD does
* Even with O_RDONLY, FreeBSD is much slower in opening (and closing) files.

I'd welcome a review of these results and comments.

_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Matthew Jacob — 2013-05-17T14:07:19

hmm. try tcopy. Can we see any complaints from /var/log/messages?
(this would be better discussed on freebsd-scsi)
_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Bob Bishop — 2013-05-17T09:22:03

Hi,

On 17 May 2013, at 01:30, Greg 'groggy' Lehey wrote:

My (dim) memory says the drive won't read at all if you get the blocksize wrong, so may be worth trying other sizes...


...but certainly try that.


There's a lot to go wrong with settings etc. See for instance: http://fixunix.com/setup/398541-dds-4-tape-drive-compatiblity.html


--
Bob Bishop
rb< at >gid.co.uk




_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Wojciech Puchar — 2013-05-17T08:10:36

all fine.


wrong. you have to specify proper block size or multiple of it.

if you don't remember block size just try.

I am not sure (not used tape for a long) but requesting larger block that 
is available should work too.

if you used dump(8) then block size is most probably 32kB.

try restore(8) or tar.
_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Matthias Andree — 2013-05-17T06:54:28

Am 17.05.2013 02:56, schrieb Zaphod Beeblebrox:

Have you used dd, tar, or pax, to actually write data?  I suppose newer
DDS drives would not actually engage the tape to their head drum until
you were actually reading/writing, in order to reduce wear and tear of
tape and heads.

Not that it helps you now:

Personally, after a few first steps with DDS1DC and DDS2, I dropped
helical scan stuff because I often found that I could read tapes only on
the same drive that had written them (I had three drives around in the
late 1990s/early 2000); no matter if the heads were cleaned or not, and
I moved on going for for optical disks and linear tape (which includues
MLR/SLR, DLT, SuperDLT, LTO; personally I used SLR2, SLR4DC and SLR6 aka
SLR24).

_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: Logging natd translations

Daniel Eischen — 2013-05-17T02:52:36


I tried the -log option to natd, but it just logged statistics,
not new connection information.  natd(8) doesn't show any other
useful options.  When I did try natd -log, that was under an
older version of FreeBSD (6.x?), but we just upgraded the system
to 9-stable and I didn't try it again.

Re: tape (sa0) on sparc64 ?

Greg 'groggy' Lehey — 2013-05-17T02:22:03

You'd expect at least a message, wouldn't you?  It's difficult to say
whether this is bitrot or taperot.

Greg
--
Sent from my desktop computer.
Finger grog< at >FreeBSD.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft MUA reports
problems, please read http://tinyurl.com/broken-mua

Re: Logging natd translations

Julian Elischer — 2013-05-17T02:04:03

why not turn on the logging on natd?

I think it has an option for logging new sessions..


_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Zaphod Beeblebrox — 2013-05-17T00:56:53

[about my tape drive not working]



 Well... therein lies my other suspicion.  I don't have any DDS4 tapes to
try writing, but the DDS3 tapes I have fail to write.

... but they don't even try.  The tape spools up when inserted and "mt
offl" works (ejects the tape) and the drive doesn't indicate any error at
this point --- but it doesn't even try to start moving for either read or
write.
_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Greg 'groggy' Lehey — 2013-05-17T00:30:59

The obvious question: can you write tapes and read them back?  My
experience with DDS tapes was of extreme unreliability.  The age
doesn't make things any easier.

Greg
--
Sent from my desktop computer.
Finger grog< at >FreeBSD.org for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft MUA reports
problems, please read http://tinyurl.com/broken-mua

Re: tape (sa0) on sparc64 ?

Zaphod Beeblebrox — 2013-05-16T23:56:14


Same result.  Besides, as far as I understand, the proper operation (if the
blocksize is too small) is to read the first $n bytes and then write them
to the output..

But same result.
_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"

Re: tape (sa0) on sparc64 ?

Bob Bishop — 2013-05-16T21:08:22

Hi,

On 16 May 2013, at 21:51, Zaphod Beeblebrox wrote:



Try bs=64k

--
Bob Bishop
rb< at >gid.co.uk




_______________________________________________
freebsd-hackers< at >freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe< at >freebsd.org"