gmane.science.linguistics.wikipedia.wikitext

Re: Collaborative Real-time Editing in VisualEditor

Ashish Dubey — 2013-04-26T06:56:59

Hi Abhishek,

I worked on this idea during the previous summer. Have a look at the
following pages to have a little insight:

https://www.mediawiki.org/wiki/User:Dash1291/GSoC_2012_Application
https://www.mediawiki.org/wiki/User:Dash1291/GSoC_status
https://www.mediawiki.org/wiki/User:Dash1291/Thoughts

So, if you've read through my application, you would know my project
revolved around implementing the Phase 1 of the whole collaboration thing.
During my timeline, I was able to implement the basic phase1, including
some basic tests. After that, I've retouched the code with occasional
refactoring to make sure the code is in a good state to be extended with
phase 2 features. Moreover, there is a tiny bit of conflict resolution code
that I've written, but its not integrated in the workflow yet. You can take
a look at the state of the code in this remote branch[1].

I would be happy to help with the code if you want to extend the work. Good
luck!

[1] -
https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extension

Collaborative Real-time Editing in VisualEditor

Abhishek Das — 2013-04-26T03:36:24

Hi!

I am Abhishek Das, a student developer from Indian Institute of Technology Roorkee, India. I have been understanding MediaWiki's code and have started contributing by solving a few bugs marked easy. Here is my user page: http://www.mediawiki.org/wiki/User:Abhshkdz

I have wide experience in building applications in PHP, MySQL & JavaScript (both client-side & server-side: Node.js). You can see my Github profile here: https://github.com/abhshkdz. 

Recently, I have been working on a lot of real-time web applications using server set events, websockets as well as Socket.IO. I have experience in building collaborative document editors. This is a video conferencing with collaborative document editing application that I built during a 24-hour hackathon and won it: https://github.com/abhshkdz/hackview. It uses webRTC for Peer-to-peer video conferencing, and collaborative doc editing over shareJS. 

Few days back, I built an open-source version of WorkFlowy (http://workflowy.com) using Backbone.js

IRC chats this week for Q&A about Lua scripting onwikis

Sumana Harihareswara — 2013-03-17T12:58:16

Hi, all!

As you might have seen on the Wikimedia tech blog
<https://blog.wikimedia.org/2013/03/11/lua-templates-faster-more-flexible-pages/>
or the tech ambassadors list
<http://lists.wikimedia.org/pipermail/wikitech-ambassadors/2013-March/000171.html>,
we enabled Lua on all Wikimedia sites last week. Lua
<https://www.mediawiki.org/wiki/Lua> is a scripting language that
enables wiki editors to write faster and more powerful MediaWiki
templates.  We're pretty excited about this, partly because of the
promise of Lua + Wikidata
<https://blog.wikimedia.org/2013/03/14/what-lua-scripting-means-wikimedia-open-source/>.
 To enable Lua templating on your own wiki, install the Scribunto
extension <https://www.mediawiki.org/wiki/Extension:Scribunto>.

If you have questions about how to convert existing templates to Lua (or
how to create new ones), we'll be holding two support sessions on IRC
this week: one on Wednesday
<http://www.timeanddate.com/worldclock/fixedtime.html?hour=02&min=00&sec=0&day=20&month=03&year=2013

Parsoid blog post & job opening

Gabriel Wicke — 2013-03-04T20:01:40

Hi,

we just published a blog post about Parsoid at

http://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with-the-web/

We are also looking for somebody to join us in our Parsoid adventure:

http://hire.jobvite.com/Jobvite/Job.aspx?j=oIsbXfw2&c=qSa9VfwQ

Cheers,

Gabriel

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: Getting started hacking parsoid

Gabriel Wicke — 2013-02-26T00:45:36

Andrew,]

thanks for your feedback!

On 02/25/2013 04:19 PM, Andrew Dunbar wrote:

I personally have not used Windows in a while, but if you have
corresponding instructions for Windows then it would be great if you
could add them. Perhaps in a subpage like
http://www.mediawiki.org/wiki/Parsoid/Windows?


these instructions were very geared to an installation with the
VisualEditor for a local wiki. I have reworded it a bit to make it clear
that you can run the web service without any config.


You can just try the built-in wikis (see example links on the API front
page) or test snippets using the forms.


This is probably no longer needed on Unix/Linux either. I have never
used it and never had problems. Removed it from the documentation.


Good point. Will add something to that effect.


That means that Windows works just as well as Linux, which is great
given that we have never tested it on Windows!

Gabriel

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.

Re: Getting started hacking parsoid

Andrew Dunbar — 2013-02-26T00:19:21

Here's some feedback from the point of view of sombody trying to use
Parsoid for the first time. Hopefully it can be taken as constructive to
help make it easier for new people to get involved, sorry if some of it
will seem nitpicky ...

- MediaWiki and node.js are cross-platform. Some of us are crazy enough to
use it on Windows but the docs are Unix-centric
- npm works great on Windows but stuff like "curlhttps://
npmjs.org/install.sh | sudo sh" does not
- Can parsoid currently be used without a local mediawiki install? this
instruction makes it seem that it cannot yet talking to people make it seem
like it should work. Confusing:

Go to the Parsoid/js/api directory and create a localsettings.js file based
on localsettings.js.example. In particular, make sure that you
useparsoidConfig.setInterwiki to point to the MediaWiki instance(s) you
want to use.

- Can you point to a remote MediaWiki instance or must it be local? Can you
leave it out?
- export NODE_PATH=node_modules is Unix-centric - what should Windo

Re: Getting started hacking parsoid

C. Scott Ananian — 2013-02-25T16:46:22

On Mon, Feb 25, 2013 at 10:46 AM, Subramanya Sastry
<ssastry< at >wikimedia.org>wrote:


The parsoid team is mostly based in California, and tend to be online PST
working hours, ie noon-8pm EST on weekdays.  That's the best time to
contact people.  Some members of the team use an IRC bot to let them catch
up on backlog when they're not online, but not all, so if you don't get a
response it's usually best to either send an email to wikitext or else just
wait until the next 'working day'.

You and Scott have similar goals then since he is interested in using

There are instructions on the wiki for cloning the git repo of parsoid and
running your own instance.  I've also successfully set up an instance on a
free appfog server; let me know if you're interested in that (and maybe I
should think about merging those patches to make it easier for others).

My embryonic offline wikibrowser is at
http://github.com/cscott/nell-wikipedia ; you can try it out at
http://nell-wikipedia.github.cscott.net, although the offline st

Re: Getting started hacking parsoid

Subramanya Sastry — 2013-02-25T15:46:44

Hi Andrew,

On 02/25/2013 06:03 AM, Andrew Dunbar wrote:

That is great.  We would love to have more people involved.


We are all around on #mediawiki-parsoid (subbu -- me, gwicke -- Gabriel 
Wicke, marktraceur -- Mark Holmquist).  Over the last couple weeks, 
we've got another active contributor cscott (C.Scott Ananian).  There 
are visualeditor folks also hanging around who participate in 
discussions once in a while.   On most weekdays, there is a fair bit of 
activity, except for occasional periods of lull.  I assume you probably 
caught a period of lull.

So, come around and say hi.


You and Scott have similar goals then since he is interested in using 
Parsoid in situations without internet or spotty internet as well, if I 
understand it correctly.   In this context, he has been doing some code 
cleanup, handling some edge cases, and contributing patches.


It is defintely faster on IRC, but wikitext-l is good for 
Parsoid-specific technical questions.

Subbu.


______________________________________

Getting started hacking parsoid

Andrew Dunbar — 2013-02-25T12:03:21

I'd like to get involved in the parsoid effort.

I've been hanging out  in the IRC channel on freenode but there's usually
just a dozen lurkers and no action.

In the mailing lists parsoid seems to be mentioned about as often in
wikitext-l and wikitech-l - which one is best to ask questions of this
nature?

I'd like to scratch my own itch rather than necessarily go after things on
the todo list and roadmap.

Basically I'm interested in what parsoid can do for parsing wikitext markup
into HTML (or other formats).

I want to use it without a mediawiki install and without an internet
connection. I see there is already some kind of support for reading in
articles from compressed dump files.

Any suggestions where I should start or where I can hang out to chat live
with people who could help getting me involved?

Andrew Dunbar (hippietrail)
_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: A grab bag of parsoid issues

Subramanya Sastry — 2013-02-21T00:05:24

Fixed in https://gerrit.wikimedia.org/r/#/c/50051/

Subbu.

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: A grab bag of parsoid issues

Gabriel Wicke — 2013-02-20T21:04:40

Scott,

On 02/19/2013 03:50 PM, C. Scott Ananian wrote:

this is really a warning about an inconsistency from the DSR (DOM source
range) calculation algorithm. We should probably change the wording to
make it less ominous.


This is https://bugzilla.wikimedia.org/show_bug.cgi?id=45208.


This is definitely a regression. Tracked in
https://bugzilla.wikimedia.org/show_bug.cgi?id=45207.


https://bugzilla.wikimedia.org/show_bug.cgi?id=45206


This is https://bugzilla.wikimedia.org/show_bug.cgi?id=45209.

Cheers,

Gabriel

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: Size: Parsoid output vs Mediawiki markup

C. Scott Ananian — 2013-02-20T19:34:36

It would be interesting to determine whether the 'Barack Obama' article is
an outlier.  It could be that the simple English wikipedia has a larger
ratio of text to markup, and thus the parsoid output is comparatively
chunkier.  "Barack Obama" may have a large amount of wikitext markup
already, so the parsoid output isn't as (comparatively) large.  If I get
some free time I'll try to do some experiments to determine what's going on.
 --scott

On Tue, Feb 19, 2013 at 7:35 PM, Gabriel Wicke <gwicke< at >wikimedia.org> wrote:

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: A grab bag of parsoid issues

Subramanya Sastry — 2013-02-20T15:54:42


Fix submitted in https://gerrit.wikimedia.org/r/#/c/50014/

Subbu.
_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: Size: Parsoid output vs Mediawiki markup

Gabriel Wicke — 2013-02-20T00:35:38

My last test after https://gerrit.wikimedia.org/r/#/c/49185/ was merged
showed a gzip-compressed factor of about 2 for a large article:

259K obama-parsoid-old.html.gz
255K obama-parsoid-adaptive-attribute-quoting.html.gz
135K obama-PHP.html.gz

We currently store all round-trip information (plus some debug info) in
the DOM, but plan to move most of this information out of it. The
information is private in any case, so there is no reason to send it out
along with the DOM. We might keep some UID attributes to aid node
identification, but there is also the possibility to use subtree hashes
as in the XyDiff algorithm to help with that.

In the end, the resulting DOM will likely still be slightly larger than
the PHP parser's output as it contains more information, in particular
about templates.

Gabriel

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: A grab bag of parsoid issues

Gabriel Wicke — 2013-02-20T00:20:53

Scott,

On 02/19/2013 03:50 PM, C. Scott Ananian wrote:

the &#10; is not important in wt2html mode, but should (ideally) be
preserved in wt2wt mode. Client DOM implementations don't preserve the
entity encoding, so we'd have to handle this with encoding-tolerant
attribute shadowing in the serializer. Handling all attribute
normalization cases explicitly would introduce a lot of complexity
however, so we prefer to hide these purely syntactic diffs in unmodified
content with selective serialization.



No normalization in the DOM makes round-tripping easier, and should lead
to the same DOM with HTML5-compliant parsers anyway.



Same issue- CSS normalization in JSDOM vs. none in Domino.

The following are really bug reports.


This is planned, but not yet implemented.


The PHP parser handles these as internal links, so we should do. We just
need to make sure that we preserve the prefix when round-tripping an
unmodified link target. The target is already shadowed, so I think the
prefix should already be prese

Size: Parsoid output vs Mediawiki markup

C. Scott Ananian — 2013-02-19T23:52:47

[Resending without the full list of articles, which caused the message to
be bounced into moderation.]

Here are the results of a quick test I ran over the weekend, comparing a
compressed excerpt from simple.wikipedia.org in mediawiki markup to the
compressed parsoid representation of the same articles.  The list of
articles is attached to this message. [Not any more.]

For the base case I used the processing pipeline for the OLPC's "Wikipedia
activity", source code at github.com/cscott/wikiserver
It begins with a hand-written "portal page", then grabs all articles within
two links of the portal page.  The original markup was taken from
the simplewiki-20130112-pages-articles.xml dump.  Templates were then fully
expanded, and just the selected articles were written.  Articles are
separated by the character 0x01, a newline, the title of the article, a
newline, the length of the article in bytes, a newline, and the character
0x02 and a newline.

For comparison, I took the list of articles included in the dump a

A grab bag of parsoid issues

C. Scott Ananian — 2013-02-19T23:50:02

For context: I've been working on replacing the html5 and jsdom modules
(which depend on the native 'contextify' module) with the pure-javascript
'domino' implementation of DOM4.  This seems to be faster, cleaner, and fix
some bug caused by jsdom's eccentric DOM handling.  Domino is (in my brief
experience) more reliable and standards-compliant.

Here's a list of issues I came across in the process:

* There were 3 new failures in wt2html tests.  (There were also some new
passes, so the number of correct tests increases on net.)  They are:

1) "expansion of multi-line templates in attribute values (bug 6255 sanity
check 2)"
For reference, this test looks like:

!! test


I'm not sure how this test ever passed in jsdom -- the inputs here are
actually identical to an HTML parser, since hex-escape decoding happens
very early.  But apparently the wikitext parser should defer processing of
the &#10 somehow?  On the domino branch our HTML serialization now uses the
upstream standard HTML5-serialization algorithm,

Re: VisualEditor update

Amgine — 2013-02-17T21:47:14

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16/02/13 12:01 PM, vitalif< at >yourcmc.ru wrote:


This has been discussed a few times: No. It means the wiki *can* work
without wikitext.

Amgine
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRIU/iAAoJECWEXpXQfyMwnqIIAKLBv+oUfto+qD2dTQZwiLSf
ufvpRMTfKidue3krCTdjSesnOl3WQYdJSqdCa+e/mgDaRAIMjm7vbx1EHyRdF5a1
YK688UmzSERSAfXXl7/UVNJBx1jsATOvHkYQXVLgGMHlLfupUWOYO6Rujh8kb3fI
gPSIW5ZUFLbMteilsG99d+fllZ5bbolD4dJnbalnvKLKK4C3yVFSmkT9PYHSNkSz
4STqVZhoYV++EkeDxNrl3o9ZN0+Ucsx7iTVAnHdqKMlUubV36Y5eaKjtDVvpIL1o
EvS+FyCc8FR4A4Yg6T5Bf+nCnq2XXVHUjtgCn7LaO/WLC+a5mZJB5m8OUeJIrqI=
=gAWL
-----END PGP SIGNATURE-----

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Re: VisualEditor update

2013-02-16T20:01:46

"Research & implement initial HTML-only wiki support (running without 
wikitext)"

Does that mean WMF wants to move away from wikitext completely? :-(

_______________________________________________
Wikitext-l mailing list
Wikitext-l< at >lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitext-l

VisualEditor update

Sumana Harihareswara — 2013-02-16T18:48:23

Earlier this month, the Visual Editor team gave an update on their
progress and their goals for 2013.  Check it out:

https://meta.wikimedia.org/wiki/File:VisualEditor-Parsoid_-_2013-02_Metrics_deck.pdf

https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/2013-02-07
has the video on Commons & YouTube.  Significantly: the Visual Editor
*should* be able to work in Internet Explorer and to handle core basic
templates, such as {{cite web}}, by 1 July 2013.

Fwd: [Wikitech-l] Extending Scribunto with new Luafunctions

Sumana Harihareswara — 2013-02-04T13:45:56

Forwarding from wikitech-l; discussion will be there.

-------- Original Message --------
Subject: [Wikitech-l] Extending Scribunto with new Lua functions
Date: Mon, 4 Feb 2013 14:46:44 +0100
From: Jens Ohlig <jens.ohlig< at >wikimedia.de>
Reply-To: Wikimedia developers <wikitech-l< at >lists.wikimedia.org>
To: wikitech-l< at >lists.wikimedia.org

Hello,

I guess this can be answered by Tim or Victor, but I'm grateful for any
pointers that can help me with a rather specific problem with Scribunto.

I'm currently working on the Wikidata project to include Lua functions
for templates that access Wikidata entities.

I've toyed around a bit and extended LuaCommon.php with a getEntities
function and a wikibase table to hold that function. Now I wonder if
there are any plans for Lua extensions outside the mw.* namespace.

I've added a wikibase.lua file and a wikibase.* namespace in Lua.
However, the way PHP and Lua play together and how Scribunto can be
extended looks a bit like black magic (which is to be expected, given
that