gmane.comp.kde.releases

Re: Change to tarball generation?

Michael Pyne — 2012-05-24T00:10:17

That's fine.


That's actually fine too, as it turns out.

As an example, try:

$ tar cf kdefoo-x.y.z.tar kdefoo-x.y.z/
$ pixz kdefoo-x.y.z.tar
# resulting in kdefoo-x.y.z.tar.xz

Because pixz is parallelized it works on whole blocks of data at a time and as 
far as I can tell makes no special provision for the last bits of compressed 
data being smaller than the block size.

With a normal tar file the decompressed data you get is:

0--------------------------------*  (where * is end of data and end of file)

With a pixz-encoded tar file the decompressed data you get is:

0--------------------------------*x$  (* is end of data, $ is end of file)

When you run a command like "tar xfJ kdefoo-x.y.z.tar.xz" everything will 
still work fine: tar knows exactly where the data should really end and will 
stop decompressing when it needs to.

When you run a pipeline like "xz --decompress kdefoo-x.y.z.tar.xz | tar xf -" 
though, there's no way to tell xz to stop decompressing early. It tries to 
write all the decompressed data to the pipe. tar still knows exactly where to 
stop, and does so at the '*', not the '$', and closes its input (a pipe!) 
early.

When xz tries to write the 'x$' (garble data) of the decompressed output it 
gets sent to a now-broken pipe, which kills xz on SIGPIPE.

Scripts trying to drive automated extraction of that data using a pipeline 
just see that an error occurred, and will therefore abort. This has affected a 
couple of distributions that are source-based, but is annoying even for those 
manually extracting to have to figure out that their tarball actually 
extracted correctly.

So the problem is only parallelizing compressors that take advantage of the 
allowance to write garbled data past the end of a file and still have the 
decompressor "figure it out". It seems pretty implausible to me that a 
parallelizing compressor would always do this, perhaps this only occurs when 
the compressor is run with tar (e.g. tar cJf) instead of as a separate step?

I hope this makes more sense.

Regards,
 - Michael Pyne

Re: Change to tarball generation?

Allen Winter — 2012-05-23T23:40:52

This whole thread is confusing me.

Maybe a command line would help?

Is this correct?
% tar cvf kdefoo-x.y.z.tar <files>
% xz kdefoo-xy.z.tar
=> resulting in kdefoo-x.y.z.tar.xz

if not, please tell us what a command line should be

I take it from mpyne's original posting that:
% tar Jcvf kdefoo-x.y.z.tar.xz <files>
isn't the way to go??

Re: Packaging 4.9: kde-base-artwork

Sebastian Kügler — 2012-05-23T21:32:40

I think it makes sense for its sheer size.

Please do keep in mind that changes in tarball layout cause additional work 
for packagers, so it's always a good idea to ask if anyone would mind (which 
then also serves the purpose that people know in advance, leading to less 
surprises and more goodwill).

Cheers,

Re: Change to tarball generation?

Wulf C. Krueger — 2012-05-23T17:28:04

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Albert et al,

On 23.05.2012 18:58, Albert Astals Cid wrote:

Even though it's somewhat coincidental ;): Thank you.

Personally, I think it would be an excellent idea to stick to the
standard xz format until the non-standard extensions that were used
for the last batch of tarballs get standardised themselves.

(Those extensions caused user-visible breakage at least on Exherbo
Linux and Gentoo Linux.)

- -- 
Best regards, Wulf
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+9HiQACgkQnuVXRcSi+5pEuQCferJgyt0sn2pgnhdUcjM5JT0X
PI0AoMcVThO4OlNyW4Fx9JabGQYV7GFx
=mm5E
-----END PGP SIGNATURE-----

Packaging 4.9: kde-base-artwork

Albert Astals Cid — 2012-05-23T16:59:58

Unless someone disagrees i'll give kde-base-artwork it's own tarball, since 
well, it's a toplevel module.

Cheers,
  Albert

Re: Change to tarball generation?

Albert Astals Cid — 2012-05-23T16:58:52

El Dimarts, 22 de maig de 2012, a les 23:48:25, Michael Pyne va escriure:

The machine i'm generating the tarballs doesn't have pixz so i'll use xz.

Albert

Change to tarball generation?

Michael Pyne — 2012-05-23T03:48:25

Hi all,

I noticed something while we were on the topic of tagging the beta tomorrow 
that I wanted to bring up, which is a concern with tarball generation. 
Specifically, the various parallizeable tarball generators (pixz, pbzip2) seem 
to generate extraneous data.

tar is smart enough to ignore this extra data, but this can affect 
decompressing our tarballs in a pipeline (i.e. xz --decompress 
kdelibs-4.foo.tar.xz | tar xf -), as tar closing its STDIN causes xz to write 
its excess data to a broken pipe.

This probably doesn't annoy a ton of different people (except for the obvious 
problem with source-based distros like Gentoo, e.g. 
https://bugs.gentoo.org/show_bug.cgi?id=410861) but if the speedup is not very 
substantial it would be better to use xz or bzip2 to avoid the problem 
entirely.

(This is done by adjusting the value of "compressors" in the pack release 
script in case you're wondering).

It might be possible to still get some concurrency benefit by batching up 
modules to "pack" and then running 4 or 8 (or however many CPUs are around) 
separate pack scripts at once, or fire off a pack while starting on tagging 
the next module, etc.

Thoughts? I'll be very clear that I don't think this should affect creating 
the beta tarballs at all but if we choose to avoid the parallelizing 
compressors hopefully that would be in time for the release candidates.

Regards,
 - Michael Pyne

Re: Time to Dump kdewebdev and kdetoys?

Nicolás Alvarez — 2012-05-23T03:00:02

2012/5/22, Alexander Neundorf <neundorf< at >kde.org>:

kdeutils indeed exists, and I'd agree with moving kfilereplace and
klinkstatus there (seems like a fitting category). However, is there
anyone interested in maintaining them?

Re: Packaging 4.9: KSecrets

Rex Dieter — 2012-05-22T23:43:40

Right, that seems to be the consensus, including it's maintainer.

Re: Packaging 4.9: kdelibs?

Andrea Scarpino — 2012-05-22T23:16:35

Whoops sorry, I read it too fast and I replied even more fast...

Packaging 4.9: nepomuk-core

Albert Astals Cid — 2012-05-22T23:07:21

Hi Vishes, how's the nepomuk situation for 4.9 Beta 1 tagging that is 
happening tomorrow?

Should I package nepomuk-core?

Albert

Packaging 4.9: KSecrets

Albert Astals Cid — 2012-05-22T23:04:31

I understand we do *not* want KSecrets in 4.9 releases, right?

Albert

Re: Packaging 4.9: kdelibs?

David Faure — 2012-05-22T22:31:41

Makes sense, for versioning.

Re: Packaging 4.9: kdelibs?

Sebastian Kügler — 2012-05-22T22:28:20

That would be consistent with what we did for 4.8, so +1.

Re: Packaging 4.9: kdelibs?

Albert Astals Cid — 2012-05-22T22:20:38

El Dimarts, 22 de maig de 2012, a les 18:02:40, Allen Winter va escriure:

Still i need to commit that version number increase somewhen if we want the 
tag to be exactly what is in the tar, and can't commit a version 4.8.80 change 
to 4.8 branch, or at least doesn't make any sense to me


I've already done that, you think i volunteered without checking the work to 
do ;-)

Albert

Re: Packaging 4.9: kdelibs?

Albert Astals Cid — 2012-05-22T22:07:09

El Dimarts, 22 de maig de 2012, a les 23:46:26, Andrea Scarpino va escriure:

You did not read the part where i can't change the version to 4.8.80 in 4.8 
branch and tag that, right?

Albert

Re: Packaging 4.9: kdelibs?

Allen Winter — 2012-05-22T22:02:40

You have to modify the version number in a "dirty" kdelibs 4.8 checkout and tar that.

Do not push any tags until the packagers have verified your tarballs are ok.

please also look at the checklist in the kde-common/release subdir.
there's a crapload of little things to do in there.

Re: Packaging 4.9: kdelibs?

Andrea Scarpino — 2012-05-22T21:46:26

If I'm not wrong Dirk releases the git tags, only when we (the packagers) 
confirm that everything's fine. So you (the KDE team) have a tag on git which 
corresponds to the tarball's content.

Also I'm not wrong there was a discussion about this.

Cheers

Packaging 4.9: kdelibs?

Albert Astals Cid — 2012-05-22T21:36:54

Soooo, now that I've bitten the bait and decided to package 4.9 beta 1 myself 
I have a question :D

kdelibs is special since we don't have a master branch to package, i need to 
package from the 4.8 branch, that's fine *but* the issue is that i have to 
increase the version number and thus I can't tag 4.8.80 since I can't commit 
the version number increase to 4.8 branch.

Ben suggests we create the 4.9 branch already. That'd go along with Martin's 
suggestion to create the 4.9 branch too.

Any comments?

Albert

Re: Delay KDE 4.9 (Was: KDE 4.9 Beta 1 Tagging)

Albert Astals Cid — 2012-05-22T21:20:42

El Dimarts, 22 de maig de 2012, a les 22:39:24, Myriam Schweingruber va 
escriure:
escriure:

Because .80 is the number we've used for all the first unstable releases 
according to http://websvn.kde.org/tags/KDE/

Albert

Re: Delay KDE 4.9 (Was: KDE 4.9 Beta 1 Tagging)

Allen Winter — 2012-05-22T21:17:56

Great.
Please do.