gmane.linux.debian.devel.general

Re: Debian documentation permalinks

Jonathan Callen — 2012-05-26T05:53:17

And here's the permalink to the above article, as it was when the
preceding post was made:
http://en.wikipedia.org/w/index.php?title=Permalink&oldid=483438630
Or, if you prefer links that don't indicate where they're really going:
http://en.wikipedia.org/w/index.php?oldid=483438630

Re: Moving /tmp to tmpfs makes it useless

Thomas Goirand — 2012-05-26T06:00:11

<disclaimer>I haven't read the actual kernel code, just making assumption
from what I generally know about swapping algorithms (eg: a long long time
ago, I worked on something like that on Atari computers...).</disclaimer>

What's going to go to the swap space are pages that have a lower rank.
There are many ways to rank a page. But these pages are generally
what hasn't been used often, and hasn't been used for a while. So let's
say you start an app, and don't use at all some of its functions, it gets
swapped out (if needed) until you start using it. This is where one can
complain about the randomness of swap access: it's not really predictable
from the user's point of view, while having a file written to /tmp is.

If you access often files in your tmpfs, of course, they wont go to the
swap, they will have a higher rank, and the kernel will decide to swap
some pages that are less often accessed. And that's when it becomes
very annoying, whatever is the swap algorithm: things go in and out
of the swap space

Re: Moving /tmp to tmpfs makes it useful

Clint Byrum — 2012-05-26T05:33:49

Excerpts from Ted Ts'o's message of 2012-05-25 18:56:55 -0700:

On laptops and other power sensitive devices, this is pretty critical.

Hypothetical: I have 2GB of RAM, and I want to watch a 50MB video file
on a connection that will take, say, 10 minutes to cache the whole thing
(and its a 10 minute video).

With a regular filesystem hosting /tmp, Every 30 seconds I will wake up
the hard disk, and write data to it. I doubt most spinning disks will
go to sleep in < 30 seconds, so this is more than 10 minutes solid of
hard disk spinning.

With tmpfs, there is no memory pressure, so my disk never even spins up
to write anything to it. If I do run into memory pressure, yes, I need
to use swap at that point. But at that point I've got a lot more than
just the disk draining power.

Re: Moving /tmp to tmpfs makes it useful

Serge — 2012-05-26T04:03:12



We're talking about *default* settings. That what makes it an issue.
It won't be an issue if most debian installations were on a small
read-only root and most of them wanted /tmp to be tmpfs because of that.
But it's not. I doubt major part of debians use r/o root and even if they
do, not all of them would like to use tmpfs for that case. Most servers
dedicate separate partitions to /tmp. I had it mount-bound to /home/tmp
in that case. There're many solutions that are fast, use fewer memory,
support quotas... Why tmpfs?

What makes things worse: there're popular programs and popular usecases
that break because of that. It may slow down or even freeze the system
because of heavy swapping. So there must be a strong good reason
surpassing these problems.

I assume people supporting /tmp on tmpfs actually use it that way.
What I don't understand is why they're arguing that "it's not that bad"
instead of just saying why it's good? Does it makes something faster?
What? How many seconds faster? Does it reduces d

Re: Exported (ba)sh functions in the environment

Philip Ashmore — 2012-05-26T03:45:30

 > Philip Ashmore <contact< at >philipashmore.com> writes:
 >
 >> Here's where I wish I was a shell script guru:
 >>    for var in `cat set.txt`; do
 >>       { if in env discard }
 >>    done
 >>    { sort offenders by decending size }
 >> Here's a summary of the ones that caught my eye. Sorry if I missed
 >> anyone out!
 > Oh.  This is smelling like artifacts of bash-completion; perhaps set is
 > dumping your completion settings.  I'm pretty sure those are not copied
 > into subshells and are only loaded in interactive shells.
 >
In my original email I ran "set > set.txt && ls -lsa set.txt".

Putting the same into a script:
    #!/bin/sh
    set > set.txt && ls -lsa set.txt

...got me 2353 bytes - I'll be a donkeys ass!

It looks like the fact that I typed it made it interactive, even though it
didn't require/allow any interaction.

Thanks,
Philip

Re: Exported (ba)sh functions in the environment

Russ Allbery — 2012-05-26T03:34:35




Oh.  This is smelling like artifacts of bash-completion; perhaps set is
dumping your completion settings.  I'm pretty sure those are not copied
into subshells and are only loaded in interactive shells.

Re: Exported (ba)sh functions in the environment

Osamu Aoki — 2012-05-26T03:32:04

Hi,

On Fri, May 25, 2012 at 08:14:28PM -0700, Russ Allbery wrote:

It was like that for me until I installed the bash-completion package -
programmable completion for the bash shell.

$ set|wc
   7355   19255  218488

HUGE!

Osamu

Re: Exported (ba)sh functions in the environment

Philip Ashmore — 2012-05-26T03:29:54

 > I'm curious why even your set of shell variables is so large, though.  My
 > environment is only 1699 bytes on a system I logged onto via ssh, and 
1998
 > on my desktop (running Xfce).  One of the biggest chunks of that is
 > LS_COLORS.
 >
Here's where I wish I was a shell script guru:

    for var in `cat set.txt`; do
       { if in env discard }
    done
    { sort offenders by decending size }

Here's a summary of the ones that caught my eye. Sorry if I missed 
anyone out!

_xspecs
__expand_tilde_by_ref()
__gdbus()
__get_cword_at_cursor_by_ref()
__git_complete_remote_or_refspec()
( lots more git functions )
__grub_dir()
_a2dismod()
_apt_file()
_debconf_show()
_dkms()
_hg()
_inkscape()
_insserv()
_module_assistant()
_openvpn()
_pbuilder()
_stg()
_svn()
_update_initramfs()
_valgrind()
_xinetd_services()

Regards,
Philip

Re: Exported (ba)sh functions in the environment

Russ Allbery — 2012-05-26T03:14:28





Only exported shell variables become part of the environment.  The rest
are local to the specific shell.  "env" will show just the environment.

I'm curious why even your set of shell variables is so large, though.  My
environment is only 1699 bytes on a system I logged onto via ssh, and 1998
on my desktop (running Xfce).  One of the biggest chunks of that is
LS_COLORS.

Re: Exported (ba)sh functions in the environment

Philip Ashmore — 2012-05-26T03:09:17

 > On 26/05/12 03:50, Philip Ashmore wrote:
 >>
 >> That's 225517 bytes that needs to be copied every time a script runs.
 >
 > Yeah that should read "every time a script or program runs."
 >
 > Philip
 >
Sorry Ben, our emails collided.

According to "man sh" (which links to the dash man page)

      set [{ -options | +options | -- }] arg ...
             The set command performs three different functions.

             With no arguments, it lists the values of all shell variables.

So are these copied every time a script runs?

Philip

Re: Exported (ba)sh functions in the environment

Philip Ashmore — 2012-05-26T02:59:58

Yeah that should read "every time a script or program runs."

Philip

Re: Exported (ba)sh functions in the environment

Ben Hutchings — 2012-05-26T02:56:56

[...]
 
No.  'env' shows the environment variables.

Ben.

Exported (ba)sh functions in the environment

Philip Ashmore — 2012-05-26T02:50:02

Hi there.

I recently had cause to search for an environment variable to see if it 
was being set.
As a result I noticed that the environment has become a bit of a dumping 
ground for
installed programs where configuration files would have been a cleaner 
option.

Looking for an override in the environment is of course a good idea, but 
setting
default values in the environment that could be

1. hard coded in the programs as defaults
2. stored in a configuration file
3. sourced from a configuration script

is a bit sloppy.

On my machine running "set > set.txt && ls -lsa set.txt" reveals that my
environment contains 225517 of "stuff" - some of it is even being taken 
up by
exported function definitions!

That's 225517 bytes that needs to be copied every time a script runs.

Regards,
Philip Ashmore

Re: Moving /tmp to tmpfs makes it useless

Serge — 2012-05-26T02:20:09


Forgot to answer that part. Sorry.


There's another feature, TMP_OVERFLOW_LIMIT to help just for that case,
it prints a warning and mounts /tmp on tmpfs if / partition gets filled.
Brilliant idea, I like it. :)


There's a difference between /tmp on a separate partition and /tmp on
tmpfs: /tmp on disk can't DoS the system, eat all the memory and make it
swapping. And regular user must not worry about that.

Imagine that you're helping your girlfriend to install debian by phone.
You can tell her "Just use the default", or you'll have to tell "make sure
you have enough swap space... and carefully unpack large archives or your
system may become slow... and don't run programs that write large files to
/tmp without configuring them... what programs? cd burners and archive
managers... and don't watch youtube videos larger than 250MB... how can
you find that out... well... just don't watch videos longer than 20
minutes..." etc. IMHO, that's what "default" settings are about. :)

Debian documentation permalinks

Philip Ashmore — 2012-05-26T02:03:06

Hi there.

First, here's what I'm talking about - 
http://en.wikipedia.org/wiki/Permalink
Unfortunately Wikipedia doesn't offer permalinks itself, so hopefully 
the above
link won't "rot".

This was initially in reference to the recent spat of comments/opinions to
"Re: Moving /tmp to tmpfs makes it useful".

It's sometimes amusing to watch as helpful enthusiasts try to unravel a 
problem.
I nearly deleted the entire thread without giving it a moments thought - 
I get
a lot of message list email.

What I noticed by its absence was that no-one linked to official Debian 
policy
detailing the choices made and their justification.

Then it struck me that if such a document existed, it would be subject 
to change
as Debian policy itself evolved, making any old links nonsensical or
misleading.

So what I'm requesting is that Debian documentation be "permalink 
friendly" i.e.
that Debian documentation pages provide permalinks.

It would be even better if they also mentioned the version(s) of Debian they
apply to.

T

Re: Moving /tmp to tmpfs makes it useless

Nikolaus Rath — 2012-05-26T01:20:07

I believe tmpfs memory is swapped out preferentially, so your scenario
doesn't have to play out like that. However, paging being a complex
process, it's not impossible either. Is that something people are
actually seeing? Because I haven't encountered this. 


Best,

   -Nikolaus

Re: Moving /tmp to tmpfs makes it useful

Ted Ts'o — 2012-05-26T01:56:55

There will be some, but really, not that much difference between going
from tmpfs to swap compared to files written to a filesystem (in both
cases the data is stored in the page cache, whether it's a tmpfs file
or an ext2/3/4 or xfs or btrfs file) in many cases.

The major difference is that tmpfs pages only get written out to swap
when the system is under memory pressure.  In contrast, pages which
are backed by a filesystem will start being written to disk after 30
seconds _or_ if the system is under memory pressure.

So if you have a super-fast networking connection, it won't matter;
the download fill the memory very quickly, at which point it will get
written to swap and/or the file location on disk at the same rate,
since you'll be able to download faster than you can save to disk, so
the network connection will get throttled due to TCP/IP backpressure
to roughly the same rate as you are writing to the HDD.

If you have a slow networking connection, it's possible that the 30
second writeback timer will

Re: Moving /tmp to tmpfs makes it useful

Miles Bader — 2012-05-26T01:46:06

Serge <sergemdev< at >gmail.com> writes:

Is that even the issue, for the most part?  My impression is that
using tmpfs is just logistically simpler and safer -- it can be
mounted very early, doesn't muck with the (possibly small or r/o) root
fs, makes it simple to limit temp space without a separate partition,
etc.

-miles

Re: Moving /tmp to tmpfs makes it useful

Serge — 2012-05-26T01:12:40

Ok, we have guesses, let's do some TESTS...



Do they become faster? My tests show they don't (see below).

I'm asking for *popular* apps, that create files in /tmp, *never put
large files* there, and become *noticeably* faster with /tmp on tmpfs?
And remember that there're popular apps that break because of that,
so to make sure it's still a good thing all 3 conditions should be met.
We're talking about defaults, not about corner cases after all...


Browsers, filemanagers, flash video, cd burners... Do you basically
suggest every debian user to think whether it's a corner case or not
and manually select TMPDIR *every time* they start browser? ;)


I believe you're not talking about some small short-lived temporary files,
because they never actually get to disk, kernel filesystem cache is smart
enough for that (unless you've changed dirty_writeback_centisecs and
dirty_expire_centisecs to some low values).

So you expect to reduce number of disk writes for *large* files
"quite a bit". I'm curious how much

Re: Moving /tmp to tmpfs makes it useless

Charles Plessy — 2012-05-26T00:36:57

Le Fri, May 25, 2012 at 12:44:03PM +0100, Roger Leigh a écrit :

Dear Roger,

thanks for your patient work.

As you explained, with latest versions of sysvinit (currently un experimetnal),
20% of the whole memory (RAM + swap) will be used for /tmp.  This will give a
couple of gigabytes in systems with 8 Gb RAM + 8 Gb swap, which may be
borderline when working with big files.  On my system at work, I often need to
sort big files, and I think that even coreutil's sort program does not check in
advance if there will be enough space in /tmp.  I often lose some time for
forgetting to set TMPDIR before running a command.

How about doing the reverse and defind the amount of memory that /tmp will
not use ?

This limit could be in the range of what is necessary to avoid a GNOME session
to be killed, and fall back to a percentage if the memory is lower than that
treshold ?  Thhe current system of using a percentage of the memory, reduces
the impact to the recommendation of increasing the swap size.  For the moment,

Re: Moving /tmp to tmpfs makes it useful

Miles Bader — 2012-05-26T00:32:50

That's not a new suggestion, it's been standard practice for nigh-on
_decades_...

-miles