gmane.linux.ubuntu.devel.distributed

DEB_VENDOR & package imports

Dmitrijs Ledkovs — 2012-05-03T13:49:19

Hello,

I am having fun times with package lp:debian/accountsservice

$ bzr cat lp:debian/accountsservice/.pc/applied-patches
Most recent Debian version: MISSING
0001-formats-locale-property.patch

0002-create-and-manage-groups-like-on-a-ubuntu-system.patch
0005-gdm_config_file_path_ubuntu.patch
0006-adduser_instead_of_useradd.patch
0007-add-lightdm-support.patch
0008-nopasswdlogin-group.patch
0009-language-tools.patch
0010-set-language.patch
0011-add-background-file-support.patch
0012-add-keyboard-layout-support.patch
0013-add-has-message-support.patch
1001-buildsystem.patch
2001-icon_reset.patch
3001-show_more_than_one_user_powerpc.patch

$ bzr cat lp:debian/accountsservice/debian/patches/series
Most recent Debian version: MISSING
0002-create-and-manage-groups-like-on-a-debian-system.patch

0005-gdm_config_file_path.patch
0006-adduser_instead_of_useradd.patch
0007-add-lightdm-support.patch
1001-buildsystem.patch
2001-icon_reset.patch
3001-show_more_than_one_user_powerpc.patch

It has both ./debian/patches/series & ./debian/patches/ubuntu.series

And upon package import it appears that ubuntu.series got applied when
importing into 'Debian' branch.

Furthermore ubuntu.series were not refreshed by the debian maintainer
upon last upload.

This results in a failure messages from $ bzr merge(-package) upon
trying to unapply patches.

I think, the lp:debian/* imports should be run with

   export DEB_VENDOR=Debian

This should make the lp:debian/package to match the source package as it
is unpacked/built on Debian.

But I'm not sure if this actually will make merging any easier.

Thoughts?

Switching over to Quantal

Barry Warsaw — 2012-04-30T15:33:16

Now that Quantal is open for development, what do we need to do to get things
switched over, and UDD all happy-like?

$ bzr branch ubuntu:precise/debootstrap precise
bzr: ERROR: Revision {package-import-GeWIH/nMZzJKr9sWaVLiSSMbcUnWK/KXJQVSFeiJYKfRRQAXIyLz2A< at >public.gmane.orgd7p4} not present in "Graph(StackedParentsProvider(bzrlib.repository._LazyListJoin(([CachingParentsProvider(None)], []))))".
$ bzr branch ubuntu:debootstrap precise
Most recent Ubuntu version: 1.0.40
Packaging branch version: 1.0.38
Packaging branch status: OUT-OF-DATE

-Barry

Importer add-import-jobs cronjob temporarily disabled

Max Bowsher — 2012-04-28T09:52:44

See https://bugs.launchpad.net/udd/+bug/990394

jubany: Importer stopped

Max Bowsher — 2012-04-26T21:38:23

I've stopped the importer. It needs to be taught about quantal before it can proceed:

Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/bin/import-package", line 7, in <module>
    main()
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 1177, in main
    only_before=options.only_before))
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 1045, in _import_package
    versions = get_versions(lp, package, extra_debian=extra_debian)
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 257, in get_versions
    vlist.sort()
  File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 265, in sort
    self.plist.sort(cmp=PackageToImport.import_order_comparator)
  File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 240, in import_order_comparator
    b_release = import_sequence_distro_releases[b.distro].index(b.release)
ValueError: list.index(x): x not in list

Importer status: back up to date, running at 8 threads

Max Bowsher — 2012-04-22T19:53:20

The importer has now caught up from the massive backlog caused by
spurious fails of just about everything.

I've reduced the concurrency to 8, to reduce the level of API requests
being directed at Launchpad by the importer's fallback re-processing of
packages that it performs whilst idle.

Max.

Stormified importer has introduced unintended change in data beingwritten to sqlite

Max Bowsher — 2012-04-21T17:44:35

The stormification has introduced an unintended change in how the data
for testaments and revision ids are stored in the sqlite DBs.

They are now being stored as binary blobs rather than strings.

The importer is currently stopped whilst I investigate further.

Max.

Importer broken again

Max Bowsher — 2012-04-21T15:03:32

I just checked up on the importer status, and found it both locked up on
DB access again, and apparently it had been running with an improper
environment setup causing it to fail to import bzr-builddeb, so it had
put ~22k packages into failure state.

I've restarted it, properly, and requeued the failures, but that means
there is now a HUGE queue.

Ideally we'd increase the concurrency to churn through the queue
quickly... but we can't because of the db bug.

I REALLY think we should be reverting the introduction of storm.

Frankly, it's not fair that another effort wanting to re-use the UDD
code should impact the operational reliability of the UDD importer to
the degree the storm introduction already has.

By all means let's land infrastructural improvements - so long as they
are reasonably safe. Now that this one has been demonstrated not so,
let's push it out of trunk until it can be made safe.

Max.

How to ??

สุทธิพงษ์ ชัยสงคราม — 2012-04-20T05:25:35

Hi, my name is Sutthipong C. from Thailand. I have the problems for build Ubuntu source code (SRC). 

Now! I have Ubuntu 11.10 (source code) and download completely from this link (http://cdimage.ubuntu.com/releases/11.10/release/source/). 

How to build from source code, Help me please.

jubany package importer stopped?

Stephanie Miller — 2012-04-16T05:40:17

Hi folks,

Our monitoring system is currently showing an alert for jubany's package
importer process, saying that no python processes were found. Restarting
mass-import doesn't seem to resolve the alert. Additionally
/srv/package-import.canonical.com/new/logs/debug_log has lots of
messages about the database being locked.

Is this the current expected state? If not, what action should I take,
if any?

Thanks,
Stephane (Canonical IS)

UDD importer making a nuisance of itself with v3 source formatbranches

Max Bowsher — 2012-04-13T00:13:22

I've just had a conversation with cjwatson and slangasek on
#ubuntu-release about the importer making a nuisance of itself by
declaring a perfectly reasonable commit to be a collision / difference,
and replacing it with one of its own.

The key pain point here is the .pc/ directory.

It's practically impossible to maintain a .pc/ directory checked into
VCS without unnatural jumping through hoops. As a result, packages being
seriously developed in bzr by humans, rather than being primarily
imported, tend NOT to have a .pc/ :

<cjwatson> patches applied doesn't have to imply .pc in vcs; it's
unfortunate that the importer took that particular decision
<cjwatson> (I've been using patches-applied-in-bzr since well before the
importer did, *without* .pc)

I think, as a short term fix, we should modify the collision-is-clean
check to ignore the absence of a .pc directory in packager-committed
revisions.

Please have a think and let me know about any objections - we should
probably treat this as urgent since it causes severe annoyance with UDD.

Max.

A huge waste of jubany diskspace...

Max Bowsher — 2012-04-10T00:11:07

pkg_import< at >jubany:/srv/package-import.canonical.com/new/tmp$ du -hsc  .
174G.

Yikes!

Anyone see any reason *NOT* to rm -rf ?

Max.

Importer stopped since Sunday?

Max Bowsher — 2012-04-10T00:08:25

It looks like someone stopped the UDD importer on Sunday?

James W. has a 'crontab -e' editor open from around that time.

Anyone know what's going on, and if it's safe to restart?

We're failing rather dismally at providing prompt imports at the moment.

Max.

bzr merge-upstream: why delete and add the same unchanged file?

Andreas Hasenack — 2012-03-20T19:20:29

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I'm experimenting with bzr merge-upstream and found what I think is an
odd behavior. I'm probably just using it incorrectly, but I can't find
out what I'm doing wrong.

I'm merging lp:landscape-client into ubuntu:landscape-client. Let's
say I'm preparing a new release.

This is the command-line:
$ bzr merge-upstream ~/canonical/source/landscape-client/trunk/
- --revision 531 --version 12.04.1

(~/canonical/source/landscape-client/trunk has lp:landscape-client)

https://pastebin.canonical.com/62652/ is the output

There are some conflicts, but I'm not worried about those for now
(unless they explain what I'm seeing).

This is my question:

$ bzr st LICENSE
removed:
  LICENSE
added:
  LICENSE

This happened to *all* files. There is not a single "modified" file in
the bzr status output.

Why is it removing and adding the same file? This file (and several
others) didn't change between ubuntu:landscape-client and
lp:landscape-client, it's exactly the same.

- -- 
Andreas Hasenack
andreas-Z7WLFzj8eWMS+FvcfC7Uqw< at >public.gmane.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9o2HwACgkQeEJZs/PdwpAqZwCgz+1yIZ1RR/sk/8oIlm9Er3Bz
olUAnAplPNp85e9gDO+FrgbF/1Nd1Xsk
=UnwB
-----END PGP SIGNATURE-----

What to do when a packaging branch is out of date

Andreas Hasenack — 2012-03-20T13:56:27

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So apparently a package update can be prepared using udd, released
into the archive, and the packaging branch still be out of date:

"""
$ bzr branch
ubuntu:landscape-client landscape-client-12.04-0ubuntu1
Most recent Ubuntu version: 12.04-0ubuntu1


Packaging branch version: 11.07.1.1-0ubuntu2
Packaging branch status: OUT-OF-DATE
Branched 40 revisions.
$
"""

It's like it was released before being committed.

What should I do if I want to prepare another fix? Wait? Ping someone
to commit and then branch again? Do it with debdiffs and not worry
about branches anymore?

I emailed ubuntu-devel-discuss and someone told me that somebody has
to manually commit to the udd branch, and that this is normal.

- -- 
Andreas Hasenack
andreas-Z7WLFzj8eWMS+FvcfC7Uqw< at >public.gmane.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9ojIsACgkQeEJZs/PdwpDzAgCgpvRePROgSwZjqwS5uWiuYCzN
WScAoPLT/aVcibCEUHEVwHE8p8o1WSGg
=VzJk
-----END PGP SIGNATURE-----

problem with recipe build

Scott Moser — 2012-03-02T22:15:24

I'm sure I'm doing something wrong, but not sure what.
I'm trying to just set up a launchpad build of two branches, one 'trunk'
that launchpad is pulling from an svn repo, and a packaging only branch.

I'm not quite sure why there would be expected to be a tag
'upstream-0.9.4+r4177' as that string is something that was created due to
this build recipe.  But the working-dir has in it a directory named
'madwifi-0.9.4+r4177' and debian/changelog in that says:
| madwifi (0.9.4+r4177-0ubuntu0+7) precise; urgency=low
|
|   * Auto build.
|
|  -- Scott Moser <smoser-GeWIH/nMZzLQT0dZR+AlfA< at >public.gmane.org>  Fri, 02 Mar 2012 16:58:43 -0500

Which seems sane to me.  The upstream source doesn't have tags named
'upstream-<version>', but only 'release-<version>'.

Help?


$ dpkg-query --show bzr-builder
bzr-builder 0.7.2-0ubuntu1

$ cat madwifi-daily.recipe
# bzr-builder format 0.4 deb-version 0.9.4+r{svn-revno}-0ubuntu0+{revno:packaging}
#/home/smoser/src/madwifi/trunk.dist
lp:~smoser/madwifi/trunk
#merge packaging /home/smoser/src/madwifi/ubuntu.dist
merge packaging lp:~smoser/madwifi/ubuntu


$ bzr dailydeb madwifi-daily.recipe working-dir
Building tree.
Retrieving 'lp:~smoser/madwifi/trunk' to put at 'working-dir/madwifi-daily-0.9.4+r{svn-revno}-0ubuntu0+{revno:packaging}'.
Merging 'lp:~smoser/madwifi/ubuntu' in to 'working-dir/madwifi-daily-0.9.4+r{svn-revno}-0ubuntu0+{revno:packaging}'.
All changes applied successfully.
Committing to: /home/smoser/src/madwifi/x/working-dir/madwifi-daily-0.9.4+r{svn-revno}-0ubuntu0+{revno:packaging}/
added debian
added debian/README.source
added debian/changelog
added debian/compat
added debian/control
added debian/copyright
added debian/dkms.conf.in
added debian/dkms_dbversion
added debian/madwifi-dkms.install
added debian/madwifi-dkms.postinst
added debian/madwifi-dkms.prerm
added debian/madwifi-tools.install
added debian/patches
added debian/rules
added debian/source
added debian/watch
added debian/patches/no-kernel-headers-required.patch
added debian/patches/series
added debian/source/format
Committed revision 1702.
bzr: ERROR: Unable to find the upstream source. Import it as tag upstream-0.9.4+r4177 or build with --allow-fallback-to-native.

UDD breakdown when building orig.tar.gz from upstream VCS

Barry Warsaw — 2012-02-15T13:21:37

I just want to put this out there for the historical record.  I think this is
a rare enough use case that UDD doesn't need to address, certainly not any
time soon, if ever.  OTOH, maybe there's an easy workaround.

I was working on an NBS for the fgfs-atlas package (LP: #903225).  The
solution was straightforward enough: upstream had all the necessary fixes in
their CVS repository, but hadn't done a release in a long time.  I twiddled
the packaging to build an orig.tar.gz from CVS, and the googlez helped find
some good general packaging information on how to do this.

Unfortunately, UDD is essentially useless here.  The problem is that after
creating the tarball from CVS, `bzr bd -S` can't be used because dpkg-source
will complain too much about deltas between the tarball and the source tree.
It'll warn about a lot of stuff, but then fail with some unrepresentable
changes to source.

I worked around this by unpacking the new orig.tar.gz and `cp -a` the debian/
directory from the precise version of the package (with my changes) over into
the unpacked tarball.  After a few rounds of tweaking, and using `debuild -S
-sa`, I had a debian/ that built locally, so I uploaded it and will let the
importer (hopefully) sort out the mess.

I still did all my changes to debian/ in a source branch though, because that
made it easier to get a diff for the linked Debian bug.

Is there's a magical udd switch or config setting that would have helped me
keep all the changes in the source branch?  It seems like this is somewhat
similar to the merge-upstream issue when upstream has a rather large released
tarball delta.

Cheers,
-Barry

xz tar imports failing?

Martin Pool — 2012-02-01T04:00:59

I haven't dug into whether these are new, or whether there is already
a bug for them, but they do seem to be new.  I see
<https://bugs.launchpad.net/udd/+bug/553668> about supporting xz in
bzr-builddeb: does that perhaps need to be deployed on Jubany?

---------- Forwarded message ----------
From: Cron Daemon <root< at >jubany.canonical.com>
Date: 1 February 2012 14:30
Subject: Cron <pkg_import< at >jubany> /usr/bin/python ${SCRIPTS_DIR}/email-failures
To: pkg_import< at >jubany.canonical.com

evince failed at 2012-01-30 13:42:01.725202 due to:
Traceback (most recent call last):
 File "/srv/package-import.canonical.com/new/scripts/bin/import-package",
line 7, in <module>
   main()
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1177, in main
   only_before=options.only_before))
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1078, in _import_package
   bstore, push, possible_transports=possible_transports)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 638, in import_package
   use_time_from_changelog=True)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1243, in import_package
   pull_debian=pull_debian)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1146, in _import_normal_package
   file_ids_from=file_ids_from)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 874, in import_upstream
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 199, in import_component_tarball
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 413, in make_pristine_tar_delta
   return make_pristine_tar_delta(dest, tarball_path)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 115, in make_pristine_tar_delta
   raise PristineTarDeltaTooLarge(stderr)
bzrlib.plugins.builddeb.upstream.pristinetar.PristineTarDeltaTooLarge:
The delta generated was too large: tar: Record size = 16 blocks
xdelta: warning: no matches found in from file, patch will apply without it
error: excessively large binary delta for
/srv/package-import.canonical.com/new/updates/evince/evince_3.3.4.orig.tar.xz
(Probably the tarball is compressed with an unsupported form of compression.)
.

xscreensaver failed at 2012-01-30 23:37:31.010788 due to:
Traceback (most recent call last):
 File "/srv/package-import.canonical.com/new/scripts/bin/import-package",
line 7, in <module>
   main()
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1177, in main
   only_before=options.only_before))
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1071, in _import_package
   possible_transports=possible_transports)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 951, in handle_collisions
   if clean_collision(importp, suite, db, temp_dir, download_dir,
name, updates_branch):
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 885, in clean_collision
   return check_same(importp, db, revid, name, updates_branch,
temp_dir, download_dir)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 852, in check_same
   importp, db, revid, name, updates_branch, temp_dir, download_dir)
 File "/srv/package-import.canonical.com/new/bzr/bzrlib/cleanup.py",
line 132, in run
   self.cleanups, self.func, self, *args, **kwargs)
 File "/srv/package-import.canonical.com/new/bzr/bzrlib/cleanup.py",
line 166, in _do_with_cleanups
   result = func(*args, **kwargs)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 872, in _check_same
   pull_debian=False)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1243, in import_package
   pull_debian=pull_debian)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1140, in _import_normal_package
   version.upstream_version)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1061, in upstream_parents
   package, pull_version.upstream_version)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 299, in version_as_revisions
   None: self.version_component_as_revision(package, version, component=None) }
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 320, in version_component_as_revision
   raise PackageVersionNotPresent(package, version, self)
bzrlib.plugins.builddeb.errors.PackageVersionNotPresent: xscreensaver
5.15 was not found in <PristineTarSource at
file:///srv/package-import.canonical.com/new/updates/xscreensaver/precise/>.

gjs failed at 2012-01-31 09:30:09.196091 due to:
Traceback (most recent call last):
 File "/srv/package-import.canonical.com/new/scripts/bin/import-package",
line 7, in <module>
   main()
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1177, in main
   only_before=options.only_before))
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1078, in _import_package
   bstore, push, possible_transports=possible_transports)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 638, in import_package
   use_time_from_changelog=True)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1243, in import_package
   pull_debian=pull_debian)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1146, in _import_normal_package
   file_ids_from=file_ids_from)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 874, in import_upstream
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 199, in import_component_tarball
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 413, in make_pristine_tar_delta
   return make_pristine_tar_delta(dest, tarball_path)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 115, in make_pristine_tar_delta
   raise PristineTarDeltaTooLarge(stderr)
bzrlib.plugins.builddeb.upstream.pristinetar.PristineTarDeltaTooLarge:
The delta generated was too large: tar: Record size = 16 blocks
xdelta: warning: no matches found in from file, patch will apply without it
error: excessively large binary delta for
/srv/package-import.canonical.com/new/updates/gjs/gjs_1.30.1.orig.tar.xz
(Probably the tarball is compressed with an unsupported form of compression.)
.

nautilus failed at 2012-01-31 15:16:23.340404 due to:
Traceback (most recent call last):
 File "/srv/package-import.canonical.com/new/scripts/bin/import-package",
line 7, in <module>
   main()
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1177, in main
   only_before=options.only_before))
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 1078, in _import_package
   bstore, push, possible_transports=possible_transports)
 File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py",
line 638, in import_package
   use_time_from_changelog=True)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1243, in import_package
   pull_debian=pull_debian)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 1146, in _import_normal_package
   file_ids_from=file_ids_from)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
line 874, in import_upstream
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 199, in import_component_tarball
   exclude=exclude)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 413, in make_pristine_tar_delta
   return make_pristine_tar_delta(dest, tarball_path)
 File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/upstream/pristinetar.py",
line 115, in make_pristine_tar_delta
   raise PristineTarDeltaTooLarge(stderr)
bzrlib.plugins.builddeb.upstream.pristinetar.PristineTarDeltaTooLarge:
The delta generated was too large: tar: Record size = 16 blocks
xdelta: warning: no matches found in from file, patch will apply without it
error: excessively large binary delta for
/srv/package-import.canonical.com/new/updates/nautilus/nautilus_3.3.4.orig.tar.xz
(Probably the tarball is compressed with an unsupported form of compression.)
.

Updates to the UPG for UDD

Barry Warsaw — 2012-01-31T23:18:08

Hi all,

I have a merge proposal to update the UDD documentation part of the packaging
guide.  I'm hoping this can be reviewed, merged, and published before my UDW
session tomorrow (it's okay if it can't).

The branch fixes some typos and formatting issues, but there are really two
substantive changes.

First, I've updated all the discussion of using `bzr merge-package` to just
use `bzr merge` as is now the case in Precise.  I do give minimum bzr and
bzr-builddeb version numbers for this, but don't dwell on the old way too
much.

Second, I've removed all the discussion of using looms to generate quilt
patches.  In going through the described procedures, I realized they just
don't work very well, and it makes things much more complicated.
Specifically, the problem comes down to when you are in the thread that
contains both the in-source change and the quilt patch, there is no way to
`quilt push` the patch, because the source tree already has the changes.

This is a problem in the non-loom approach (which I describe better now, and
have tested), but it's easier because you can just shelve or revert the
in-source change once you've imported the quilt patch.  I've actually been
using this fairly successfully for a while now.  While I still like the idea
of using looms instead of quilt, and am a little sad to let go of this, for
now, I think it's clearer to avoid the whole thing.

So, I hope these changes simplify the UDD workflow.  I've run through and
tested them with various packages so I'm pretty sure the instructions are
accurate.  Reviews are welcome of course!

https://code.launchpad.net/~barry/ubuntu-packaging-guide/udd-update/+merge/90976

Cheers,
-Barry

P.S. See you at 1900 UTC in #ubuntu-classroom tomorrow.

Improved quilt patch handling

Jelmer Vernooij — 2012-01-19T00:30:46

bzr-builddeb 2.8.1 has just landed on Debian Sid and Ubuntu Precise. 
This version contains some of my improvements from late last year for 
the handling of quilt patches in packaging branches. Most of these 
improvements depend on bzr 2.5 beta 5, which is also in Sid/Precise.

The most relevant changes (enabled by default) are:

  * 'bzr merge-package' is now integrated into 'bzr merge' (it's just a 
hook that fires on merges involving packages)
  * patches are automatically unapplied in relevant trees before merges
  * before a commit, bzr will warn if you have some applied and some 
unapplied quilt patches

Furthermore, you can now specify whether you would like bzr to 
automatically apply all patches for stored data and whether you would 
like to automatically have them applied in your working tree by setting 
'quilt-tree-policy' and 'quilt-commit-policy' to either 'applied' or 
'unapplied'.  This means that you can have the patches unapplied in the 
repository, but automatically have them applied upon checkout or update. 
Setting these configuration options to an empty string causes bzr to not 
touch your patches during commits, checkout or update.

We've done some testing of it, as well as running through a package 
merge involving patches with Barry, but none of us do package merges 
regularly. If you do run into issues or if you think there are ways we 
can improve the quilt handling further, please follow up to this email 
or file a bug report against the UDD project 
(https://launchpad.net/udd/+filebug).

Caveats:

  * If there are patches to unapply for the OTHER tree, bzr will 
currently create a separate checkout and unapply the patches there. This 
may have performance consequences for big packages. The best way to 
prevent this is to set 'quilt-commit-policy = unapplied'.
  * 'bzr merge' will now fail if you are merging in a packaging tree 
that is lacking pristine tar metadata; I'm submitting a fix for this, 
but it's not in 2.8.1.
  * if you set 'quilt-commit-policy' and 'quilt-tree-policy' but have 
them set to a different value, bzr will consider the tree to have changes.

To disable the automatic unapplying of patches and fall back to the 
previous behaviour, set the following in your builddeb configuration:

'quilt-smart-merge = False'

Cheers,

Jelmer

Moving udd to django

James Westby — 2011-12-10T02:10:48

Hi,

I think there are a few reasons that we should consider moving udd to
django (more on what this actually means later.)

  1. We want to have remote commands so that we don't have to ssh in and
  run scripts for common tasks

  2. There are scripts in cron that generate html pages that are
  served. It would be nice to have these generated on request so that
  there is no latency.

  3. It would also allow for starting to move udd to an SOA, or at least
  make it easier.

  4. It would be nice to have a query builder, rather than all the
  hand-written sql.

So, what exactly do I mean by moving to django?

  * Moving all database access to go through django's ORM.

  * Moving all HTML generation to be done in django views.

This has fairly wide-ranging consequences though, and obviously needs to
be broken down in to manageable steps.

Here's my idea for what those steps should be.

1) Install python-django in production
   - So that it is there when it is needed.

2) Create a django project in the udd codebase.
   - Providing the skeleton to use, but not actually changing any
   behaviour

3) Write django management commands for each current script that call
the same code.
   - This gets them running in the django env, but without changing
   anything else.
   - At this point we'll want to deploy, and change the init script and
   cronjobs to run the new scripts.

4) Configure the django project to use multiple databases corresponding
to the existing database.
   - At this point we'll want to deploy and run some sanity checks to
   make sure that django can read the dbs.

5) Pick a db table and write a django model to match. Convert users of
that table to call through the query builder. Repeat.
   - I think this will be ok table-at-a-time, but we should obviously
   test well.

6) At this point we'll be using django just as an ORM, but can turn on
the admin site if we want.

7) We can also start to write django views to replace the
html-generating scripts. Once they are written we can deploy and request
that apache actually run django and serve the views, and then turn of
the scripts.

8) Then we'll be able to start adding forms to the views to accomplish
common tasks.

That's a fairly long road, but it is incremental, and the changed code
deployed at each point can be fairly small.

What do you think? Is this worth doing?

Thanks,

James

Using lp:udd beyond its original purpose

Jonathan Lange — 2011-12-02T14:18:19

Hello,

We're in the process of deploying lp:udd to do something that has very
little to do with Ubuntu distributed development. We're using it to
maintain a local database mapping from object files to symbols, for
our own nefarious purposes[1]. It works well for this, but there are
some changes that we will want to make soon.

There are two big changes that aren't necessary for us, but would help a lot:
  1. Decouple lp:udd from the package-import.canonical.com production deployment
  2. Split the package scanning and the branch importing parts of
lp:udd into separate projects

It sounds scary when I write it like that, but I don't think it's
actually that much code.

Neither of these things has to be done – we'll figure something out –
but I reckon these changes will result in less code that's better
tested.

What say you?

jml

[1] https://wiki.ubuntu.com/AutomagicBinaryPackaging