gmane.org.wikimedia.xmldatadumps

Re: How to create pages-articles.xml.bz2

Ariel T. Glenn — 2013-06-14T06:24:04

Στις 13-06-2013, ημέρα Πεμ, και ώρα 12:26 +0200, ο/η Christian
Aistleitner έγραψε:

You don't need to pipe the output to anything; you can specify that you
want bzip2 when giving the --output option to dumpBackup.php, as
follows:

php ./dumpBackup.php --full --output=bzip2:/path/to/output/fulls.xml.bz2

This is the command for one-pass dumps, as Christian noted above.

Ariel


_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: empty enwiki incremental dumps

Ariel T. Glenn — 2013-06-13T19:29:53

A colleague restarted replication (it turns out there was no other
underlying problem) so for the next few days we might see some very
large 'incrementals' :-)

Ariel

Στις 13-06-2013, ημέρα Πεμ, και ώρα 16:04 +0300, ο/η Ariel T. Glenn
έγραψε:

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: empty enwiki incremental dumps

Ariel T. Glenn — 2013-06-13T13:04:50

The database server which received the queries is not doing replication.
Over the next couple of days we should have this straightened out.

Ariel

Στις 13-06-2013, ημέρα Πεμ, και ώρα 15:42 +0300, ο/η Ariel T. Glenn
έγραψε:



_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: empty enwiki incremental dumps

Ariel T. Glenn — 2013-06-13T12:42:41

This is being investigated.

Ariel

Στις 06-06-2013, ημέρα Πεμ, και ώρα 10:06 -0700, ο/η Xiliang Zhong
έγραψε:




_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

wikidatawiki dumps blah blah

Ariel T. Glenn — 2013-06-13T11:27:09

As usual I am running a couple steps manually, which is why it looks
like nothing is happening and that the dump is failed.  Don't worry, in
a day or two the last bits will magically show up.

Ariel



_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: How to create pages-articles.xml.bz2

Christian Aistleitner — 2013-06-13T10:26:35

Hi,

On Thu, Jun 13, 2013 at 09:33:27AM +0200, witzman-HFF/x49lr9I3+Z6f+qA8vg< at >public.gmane.org wrote:

It looks like, you effectively created a pages-articles.xml.tar.bz2
(xml /in a tar file/, which is bzip2 compressed), instead of a
pages-articles.xml.bz2.

You can use the bzip2 program directly (see "man bzip2") to compress
the .xml, instead of using it through "tar". That should give you a
proper .bz2 file.



For wikimedia, they are generated using
https://gerrit.wikimedia.org/r/#/admin/projects/operations/dumps
(use the "ariel" branch)

First stubs (Xml structure without the actual Wikitext) are
generated. Afterwards, those stubs get filled with the Wikitext.

But dumpBackup.php should give you a working xml as well (see
above). If it does not, that is a bug from my perspective.


Best regards,
Christian

How to create pages-articles.xml.bz2

2013-06-13T07:33:27

hi,

I run www.ameisenwiki.de and i want to create dumps for wikitaxi. i 
need the pages-articles.xml.bz2 format.

currently i try this with php dumpBackup.php --full > 
/var/www/wiki/dump/pager-articles.xml and create the .bz2 file 
afterwards.

wikitaxi is not able to import - parser error if i use dumpgenerator.py

i also get an incompatible xml Which tool is used to create these 
exports?

You can have a look on the Dumps here:
http://www.ameisenwiki.de/dump/

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

empty enwiki incremental dumps

Xiliang Zhong — 2013-06-06T17:06:08

Hi,

I tried to access recent enwiki incremental dumps and noticed that
they contains no updates.

Eg below texts were all I got after unzipping the dump file.  I tried
a few enwiki incremental dumps of a few days ago and got the same
thing.
Can you please take a look and fix it ? thanks.
Xiliang

http://dumps.wikimedia.org/other/incr/enwiki/20130605/enwiki-20130605-pages-meta-hist-incr.xml.bz2

- <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.8/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.8/
http://www.mediawiki.org/xml/export-0.8.xsd" version="0.8"
xml:lang="en">
- <siteinfo>
  <sitename>Wikipedia</sitename>
  <base>http://en.wikipedia.org/wiki/Main_Page</base>
  <generator>MediaWiki 1.22wmf5</generator>
  <case>first-letter</case>
- <namespaces>
  <namespace key="-2" case="first-letter">Media</namespace>
  <namespace key="-1" case="first-letter">Special</namespace>
  <namespace key="0" case="first-letter" />
  <namespace key="1"

Re: WP-MIRROR 0.6 Released

Ariel T. Glenn — 2013-06-03T14:40:11

Στις 03-06-2013, ημέρα Δευ, και ώρα 10:22 -0400, ο/η wp mirror έγραψε:
Sounds great! Can you give us some benchmarks?  I'm particularly
interested in length of time with the old and nw versions of your
package for the various stages of setting up a dump of current pages for
the English language Wikipedia, on whatever hardware you are using for
testing.

Ariel


_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

WP-MIRROR 0.6 Released

wp mirror — 2013-06-03T14:22:53

Dear list members,

I am pleased to announce the release of WP-MIRROR 0.6.

The main design objective was this:  PERFORMANCE.  WP-MIRROR 0.6 now
builds the `enwiki' (which is the most demanding case) with 80% less
time and 75% less memory than v0.5.

Feature:  One new feature was added.  WP-MIRROR 0.6 can now mirror
wikis from most other WMF projects (e.g. wikibooks, wiktionary, etc).

Reliability:  Downloads are now performed with the aid of `wget' which
has an automatic restart feature.  This virtually eliminates the
problem of partial downloads.

Images:  WP-MIRROR 0.6 makes use of image dump tarballs found at
<http://ftpmirror.your.org/>.  WP-MIRROR 0.6 then does a thorough job
of identifying image files missing from the tarballs, and downloads
them efficiently using HTTP/1.1 persistent connections.

Packaging:  The DEB package for WP-MIRROR 0.6 should work
`out-of-the-box' with no user configuration for the following
distributions:

o Debian GNU/Linux 7.0 (wheezy)
o Ubuntu 12.10 (quantal)

Virtual Hosts

Re: [Pkg-mediawiki-devel] Name for DEB package

Thorsten Glaser — 2013-06-03T12:14:58


The rule of separation here is that it has separate upstream.


This also means that the conversion utilities do not require
anything from the MediaWiki packages themselves (like the
extensions do), and that they’re only useful for a small
subset of MediaWiki users (those that indeed use MySQL instead
of a database). Which confirms that.

In case you *do* need something to be changed in the core
mediawiki* packages in Debian, of course this is the right
place to go. (And maybe one of the other readers will show
interest in those packages – just I don’t.)

bye,
//mirabilos

Re: [Pkg-mediawiki-devel] Name for DEB package

Thorsten Glaser — 2013-06-03T09:39:56


For what’s it worth, I ACK’d your mails, but if you’re going to
write here more often you should maybe subscribe to the list.
And use a realname ☺


That’s probably correct.


It’s for MySQL instead of a database, so I’m neither personally
nor professionally interested, but maybe someone else. Otherwise…


… this is an important first step for…


… this. Indeed, as this package does not have anything to do
with MediaWiki AFAICT from a quick glance, not being an extension
or somesuch, just submit it using the normal Debian ways.

bye,
//mirabilos

Re: [Pkg-mediawiki-devel] Name for DEB package

Ariel T. Glenn — 2013-06-03T10:41:43

Στις 03-06-2013, ημέρα Δευ, και ώρα 11:39 +0200, ο/η Thorsten Glaser
έγραψε:

<snip>

The main utility is designed to read XML dumps that conform to the
Mediawiki export schema [1] and write sql for importing into three MW
tables.  If that should be treated as separate from MediaWiki for .deb
purposes, that's fine, there's a Java MW XML->MySQL conversion utility
with somewhat different functionality that would be nice to get packaged
up too, and we can follow the same route for both of them.

Thanks,

Ariel

[1] http://www.mediawiki.org/xml/export-0.8.xsd



_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: Help needed for Volunteer Uganda project - Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Federico Leva (Nemo — 2013-06-03T10:20:37

Richard Ive, 03/06/2013 12:06:

You can also use the kiwix server feature from a central machine to all 
machines connected via LAN if disk space is an issue more than 
connection; WMFR has a whole african program on this.

Nemo

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: Help needed for Volunteer Uganda project - Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Richard Ive — 2013-06-03T10:06:13

In all honesty, I didn't know about it until now.

Everything else we are using is web biased (Khan Academy lite, ebooks and
emedia), so for our model the Wikipedia website works best. I would guess
it is cheaper to buy a £300 desktop with 2TB for the wiki Mysql database,
than getting 15 SD cards > 16GB (I'm guessing at disk usage)?

Thanks for pointing this out though! I'll definitely consider using it in
the future.

Kind regards,
Richard Ive



On 3 June 2013 10:17, Federico Leva (Nemo) <nemowiki-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Help needed for Volunteer Uganda project - Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Federico Leva (Nemo — 2013-06-03T09:17:55

Richard Ive, 02/06/2013 12:38:

Why aren't you using Kiwix? Reportedly, it even runs standalone on a 
Raspberry Pi without problems.

Nemo

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: Help needed for Volunteer Uganda project - Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Richard Ive — 2013-06-03T08:59:57

Thanks for confirming 13mil is enough.

I didn't have the extension installed. It looks a lot better now. Thank
you!

There are still a few weird looking things around templates, but I'm
importing enwiki-20130403-templatelinks.sql now so hopefully that should
fix things.

Thanks again.


On 3 June 2013 09:07, Ariel T. Glenn <ariel-AeOJrEpdGNeGglJvpFV4uA< at >public.gmane.org> wrote:

Re: Help needed for Volunteer Uganda project - Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Ariel T. Glenn — 2013-06-03T08:07:06

Στις 02-06-2013, ημέρα Κυρ, και ώρα 11:38 +0100, ο/η Richard Ive έγραψε:

...

Some pages are not included in this dump, e.g. user and user talk pages.
You don't need those for display of the articles.  A quick count of the
titles in the stubs file shows 13 million or so pages so you're all
good.


The templates are included in the pages-articles dump.  Are you sure you
have the ParserFunctions extension enabled?

Ariel



_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Help needed for Volunteer Uganda project -Importing enwiki-20130403-pages-articles-multistream.xml.bz2

Richard Ive — 2013-06-02T10:38:09

Hi all,

I am helping the charity Volunteer Uganda set up an offline eLearning
computer system with 15 Raspberry Pi's and cheap desktop computer for a
server. Server stats:

   - 2TB disk
   - 8GB DDR3 ram
   - 3ghz i5 quad core.

I am trying to import enwiki-20130403-pages-articles-multistream.xml.bz2
using mwdumper-1.16.jar, but I have a few questions.


   1. I was originally using a GUI version of mwdumper-1.16.jar, but that
   errored out a few time with duplicate pages so I decided to use the
   pre-built one recommended on the media wiki page. Having looked at the
   stats on Wikipedia I can see that there are roughly 30 million pages,
   however I have found this morning that mwdumper-1.16.jar has finished (no
   errors) with roughtly 13.3 million pages. Without any errors I assumed that
   it had finished, but I appear to be 17 million pages short?
   2. The pages that have imported are missing templates. Is there another
   XML file that I can import which will add the missing templates? As the

Re: Tools to help import MediaWiki XML dumps into database

Dr. Kent L. Miller — 2013-06-01T01:57:04

Dear Ariel,
 I have the Debian Intent-To-Package (ITP) bug number: #710584.
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710584>.

Sincerely Yours,
Kent

+------------------------------------------------------------------------+
pub  1024D/359E5142 2008-09-01       GPG key available on pgpkeys.mit.edu
      Key fingerprint = 8D4F 4485 7F7D 5406 230C  9749 B821 2572 359E 5142
uid  Dr. Kent L. Miller <kent.l.miller-VL1tVIhH15PQjUSlxSEPGw< at >public.gmane.org>
+------------------------------------------------------------------------+


On Fri, 31 May 2013, Ariel T. Glenn wrote:

Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Re: Tools to help import MediaWiki XML dumps intodatabase

wp mirror — 2013-06-01T03:47:08

Dear Ariel,

The package is now posted on:

<https://mentors.debian.net/package/mediawiki-mwxml2sql>.

Sincerely Yours,
Kent

On 5/31/13, wp mirror <wpmirrordev< at >gmail.com> wrote:

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l