gmane.science.linguistics.wikipedia.technical

Major take-away from architecture meeting at Hackathon: RFCs will be reviewed by architecture group by Wikimania

Mark A. Hershberger — 2013-05-25T13:10:42

For those of you not at the hackathon, there is one very major result
from the meeting that I wanted to publicize.  (There are other action
items which I'll let other people expand but this one seems most
important to me.)  If I misunderstood or mis-state something, Robla and
others can correct me.

Our RFC process has been broken till now.  WMF's architecture group has
just committed to review and respond to all RFCs by WikiMania.  This
will probably involve clarifying the process that RFCs are reviewed and
providing better documentation for architectural and design direction.

I expect there will be more coming out of this meeting, but I wanted to
post this note so I would remember to ask Brion about the RFCs he has
reviewed next Friday.

Mark.

Re: pt.wp Mentorship Program

Federico Leva (Nemo — 2013-05-25T08:44:42

I don't know what you mean by "similar program", but of course the 
Germans have all their local groups, courses, workshops & co. to "train 
the workforce". For instance https://de.wikipedia.org/wiki/Wikipedia:WikiCon

Nemo

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Aaron Schulz — 2013-05-25T05:44:04

Indeed, the queue cannot use memcached. Redis will trivialize the time spent
on actually queue operations, which could help if that is a bottleneck for
job runners. If the actual jobs themselves are slow, of course it won't help
too much.

Have you already tried setting the job run rate to 0 and is a background
script instead?



--
View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005693.html
Sent from the Wikipedia Developers mailing list archive at Nabble.com.

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Jamie Thingelstad — 2013-05-25T00:03:20

I can see how memcached and redis would have similar performance for sessions. Is that also true for job queue? I was assuming that job queue could move from database to redis, but not to memcached. Is that a correct assumption?

The main reason I could see moving sessions into redis would be to not manage both redis and memcached, assuming that redis is the key to making the job queue faster. 

--
Jamie Thingelstad
jamie< at >thingelstad.com
mobile: 612-810-3699

On May 24, 2013, at 3:07 PM, Aaron Schulz <aschulz4587< at >gmail.com> wrote:

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Aaron Schulz — 2013-05-24T20:07:04

Note that if you already use memcached for the main cache, there isn't really
any reason to switch to redis unless you need replication or persistence.

Anyway, to use it for sessions, if you had $wgSessionCacheType explicitly
set to something, then you'd need to change that too (like to 'pecl-redis').
In any case, it doesn't hurt to be explicit. This all assumes that
$wgSessionsInObjectCache = true as well.



--
View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005665.html
Sent from the Wikipedia Developers mailing list archive at Nabble.com.

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Aaron Schulz — 2013-05-24T20:00:02

2.2.2 of the extensions works for me. I downloaded it from source and
compiled it.

The redis server itself will need to be 2.6 or higher for the job queue.

Looking around, I forgot to mention that JobQueueRedis was actually removed
from 1.21 (though it's in master and will be in 1.22).



--
View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005664.html
Sent from the Wikipedia Developers mailing list archive at Nabble.com.

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Jamie Thingelstad — 2013-05-24T19:57:08

If you implement this, is there any use for memcached still? I assume PHP also needs to be told that sessions are in redis?
Jamie Thingelstad
jamie< at >thingelstad.com
mobile: 612-810-3699
find me on AIM Twitter Facebook LinkedIn

On May 24, 2013, at 2:44 PM, Aaron Schulz <aschulz4587< at >gmail.com> wrote:


_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Jamie Thingelstad — 2013-05-24T19:50:36

Awesome! Thank you!

Do you know the version required for these? Thinking maybe I could writeup a page on mw.o documenting the process of putting redis in place for these.
Jamie Thingelstad
jamie< at >thingelstad.com
mobile: 612-810-3699
find me on AIM Twitter Facebook LinkedIn

On May 24, 2013, at 2:44 PM, Aaron Schulz <aschulz4587< at >gmail.com> wrote:


_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Information on MW and Redis?

Aaron Schulz — 2013-05-24T19:44:58

To use redis as a cache you can have something like:

// requires phpredis extension for PHP
$wgObjectCaches['pecl-redis'] = array(
'class'   => 'RedisBagOStuff',
'servers' => array( '127.0.0.1:6379' ),
);
$wgMainCacheType = 'pecl-redis';

This would also require that the redis server would have allkeys-lru for its
eviction policy in redis.conf.

To use redis for a jobqueue, one can have something like:

// requires phpredis extension for PHP
$wgJobTypeConf['default'] = array(
'class'          => 'JobQueueRedis',
'redisServer'    => '127.0.0.1:6379',
'redisConfig'    => array(),
'claimTTL'       => 3600
);

This works best if the redis server uses rdb snapshots and/or
append-only-file logging in redis.conf so that jobs are lost with power
outages or restarts.



--
View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005660.html
Sent from the Wikipedia Developers mailing list archive at Nabble.com.

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Information on MW and Redis?

Jamie Thingelstad — 2013-05-24T19:39:23

Hello all,

I keep seeing references in WMF documents to using Redis for session storage and as the storage for the job queue going forward. However, LocalSettings doesn't have any references to Redis. It looks like the session changes for Redis were in 1.20, and I thought that the change for the job queue to be backed by Redis was coming in 1.21. It also seems that Notifications (Echo) may require Redis, but that's not real clear.

I'm wondering what the general status of this is? As a 3rd party mediawiki admin, I'm eager to get redis going primarily for the job queue improvements, but with nothing noted in local settings documentation I'm wondering what the plan is for these redis based features? Will $wgMainCacheType be getting a CACHE_REDIS option?

Jamie Thingelstad
jamie< at >thingelstad.com
mobile: 612-810-3699
find me on AIM Twitter Facebook LinkedIn

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: database encoding for field with mathematicalexpressions

Matthew Flaschen — 2013-05-24T19:37:57

The Gerrit for this is https://gerrit.wikimedia.org/r/#/c/61987/

Matt Flaschen

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Hackathon idea: How about a live bug triage and code review hour ?

Andre Klapper — 2013-05-24T14:25:54

I'd be in and available, if somebody defines a room & time for it.
(I'm also in for any Bugzilla or bug report questions in general.)

andre

Re: Recursive merging enabled for Gerrit repositories

Martijn Hoekstra — 2013-05-24T11:36:48

awesome!


On Thu, May 9, 2013 at 11:42 PM, Chad <innocentkiller< at >gmail.com> wrote:

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Publish-staying-in-editmode feature for WikiEditor

2013-05-24T11:26:20

Hello!

I have implemented an idea for WikiEditor extension: replace 
"step-by-step publish" feature with another one - "publish staying in 
edit mode via AJAX". You can see a demo at http://wiki.4intra.net/ if 
you want. It works simply by sending an API save article request while 
NOT closing the article being edited. Also it handles section edits 
correctly via re-requesting section content after editing, so you'll 
stay with consistent edit form even if you add sections.

The idea is to give authors the ability to save intermediate results.

My question is - does anyone really need "step-by-step publishing" 
feature that is in WikiEditor? I think it's useless because it just 
duplicates the existing functionality, just submits the form using 
normal POST request, and makes editing harder as you have to do more 
clicks. I would submit a patch to Gerrit if you're interested in 
replacing it with "publish-staying-in-editmode".

Re: database encoding for field with mathematicalexpressions

Kevin Israel — 2013-05-24T08:07:53

If you use MySQL, when you installed MediaWiki (or created the table),
did you choose the "UTF-8" option instead of "binary"? The underlying
MySQL character set is "utf8"[1], which does not support characters
above U+FFFF (four-byte characters).

This is mentioned in the web installer (message 'config-charset-help'):


MySQL 5.5 did introduce a new "utf8mb4" character set, which does
support four-byte characters; however, MediaWiki does not currently
support that option (now filed as bug 48767).

The WMF of course has to use the 'binary' option (actually, UTF-8 stored
in latin1 columns, as mentioned in bug 32217) to allow storage
of all sorts of obscure characters from different languages.

utf8_encode()/utf8_decode() work around the problem because they replace
byte values 80 to FF with two-byte characters from U+0080 to U+00FF,
(encoded as C2 80 to C3 BF) and the 'utf8' option does allow those
characters.

[1]: https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8.html
[2]: http://en.wikipedia.org/wiki/Mapping_of_Unicode_character_planes

Re: Need help with a Jenkins PHPUnit build failure

Brad Jorsch — 2013-05-24T06:45:41

On Fri, May 24, 2013 at 2:29 AM, Arthur Richards
<arichards< at >wikimedia.org> wrote:

Actually, it depends on how you run PHPUnit on your extension.

If you do 'php phpunit.php /path/to/your/extension/', it will add any
file ending in "Test.php" that seems to contain a unit test.

If you do 'php phpunit.php --filter ExtensionName', it will use the
UnitTestsList hook.

database encoding for field with mathematicalexpressions

2013-05-24T03:31:18

Hi,

I'm a testing a new rendering option for the <math /> element and had
problems to store MathML elements in the database field
math_mathml which is of type text.
The MathML elements contain a wide range of Unicode characters like the
INVISIBLE TIMES that is encoded as 0xE2 0x81 0xA2 in UTF-8 or even 4 byte
chars like MATHEMATICAL BOLD CAPITAL A  0xF0 0x9D 0x90 0x80 .
In some rar cases I had problem to retrieve the stored value correctly from
MySQL.
To fix that problem I'm now using the PHP functions utf8_encode /decode to
which is not a very intuitive solution.
Do you know a better method to solve this issue without to change the
database layout.

Best
Physikerwelt




_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Need help with a Jenkins PHPUnit build failure

Arthur Richards — 2013-05-24T00:29:31


After some digging it became obvious that adding classes that are not test
cases using 'UnitTestsList' just doesn't work since PHPUnit will only
actually add the files that contain test cases. I instead tried using
$wgAutoloadClasses inside of efExtMobileFrontendUnitTests() to load the
extra helper class, but that didn't seem to work either :( Ultimately, I
rewrote how the tests work and merged the helper class into the same file
containing the test case that relied on it. But for the future, is there a
better way to include arbitrary, non-PHPUnit test case classes in test runs?

pt.wp Mentorship Program

Alchimista — 2013-05-23T23:18:09

On pt.wikipedia there is a lack of local tech people who can do simple
things like fixing a .js gadget (or creating one for a specific and local
utility), or a simple bot script, and when it comes to lua modules, it's
easy to guess... After a discussion, we've decided to implement our own
local version of wmf mentorship programs like LevelUp, but instead of
competing with them, complement where we can.[0] It basically intents to
fill the gap between those who would like to know some js to deal with
gadjets or lua to create nice modules, or even python to write some basic
bots, and then those who wish to go further, they'll always have global
programs.

We'll basically start slowly on Python, we've created some doc, mostly to
introduce and inform where to find the info, both on web or on mw. We're
trying not to replicate information, just stick to fill the blanks, and
mapping the disperse info (Gil and Sumana had gave a lot of good advices,
by the way).

So now that we're trying to get it to work, is there any similar program on
other wm project to share experiences, or specially for those who had been
mentors on global projects, any suggestion or advice? As far as i know,
none of us have participated on a similar project, so we're accepting free
( a(s|nd) free bee r) suggestions :D


Now, a second question. We're starting on python, more precisely python +
pywpbot, and i was wondering what would be the most appropriated place
where this scripts could be tested. It could be on pt.wp, but people can be
afraid of breack something, at least when i cooked my first py script, i
only got confident once i got the bot flag on test, could test it on a few
pages there. Could http://test.wikipedia.org/ this purpose, or is there any
other place where they wount feel the "elephant in a *porcelain store"
syndrome? (This is a common Portuguese *saying, i believe you got the
message)


Cheers,

Re: Need help with a Jenkins PHPUnit build failure

Matthew Flaschen — 2013-05-23T21:42:23

My guess is you need to add it to $autoloadClasses.


I don't know if Jenkins uses that.  Moreover, it doesn't seem correct to
add it there, since that file doesn't actually have tests in it.

Matt Flaschen

_______________________________________________
Wikitech-l mailing list
Wikitech-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Need help with a Jenkins PHPUnit build failure

Arthur Richards — 2013-05-23T20:59:38

In trying to submit https://gerrit.wikimedia.org/r/#/c/63907/18, I get a
Jenkins build failure due to PHPUnit failing, due to a Fatal being thrown:
https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-testextensions-master/898/console

The fatal:

PHP Fatal error:  Class 'MFMockRevision' not found in
/srv/ssd/jenkins/workspace/mwext-MobileFrontend-testextensions-master/extensions/MobileFrontend/tests/MobileContextTest.php
on line 500


The thing is, 'MFMockRevision' should be made available by it's file
being included in efExtMobileFrontendUnitTests() (our hook handler for
UnitTestsList.


Unit tests execute fine for me and at least one other person on the
mobile team. I do not want to manually merge that patchset since I
imagine it would cause build failures on subsequent patchset +2's.
Anyone know what might be going on?