gmane.ietf.charsets

RE: New charset registry entry for iso-8859-11, anybody?

Shawn Steele — 2012-10-01T21:00:50

Windows/.Net treat a request for "iso-8859-11" as "windows-874", same as "tis-620".  Technically they aren't quite the same thing.  It might be nice if whatever happened acknowledged the variants, as we've done for some other code pages.  Windows' names though, would break existing data if they changed.

-Shawn

-----Original Message-----
From: Ned Freed [mailto:ned.freed< at >mrochek.com] 
Sent: Monday, October 1, 2012 8:05 AM
To: "Martin J. Dürst"
Cc: ietf-charsets
Subject: Re: New charset registry entry for iso-8859-11, anybody?

Speaking entirely as a contributor, so am I. We should not be promulgating ISO nomes that don't match the actual charset content.

Exactly why I think 2 is the better option.

Ned

Re: New charset registry entry for iso-8859-11, anybody?

Ned Freed — 2012-10-01T15:04:38













Speaking entirely as a contributor, so am I. We should not be promulgating
ISO nomes that don't match the actual charset content.


Exactly why I think 2 is the better option.

Ned

New charset registry entry for iso-8859-11, anybody?

Martin J. Dürst — 2012-10-01T10:47:12

Dear Charset Experts,

Behind the scenes, there have been some discussions about adding an 
entry for ISO-8859-11, Latin/Thai.

However, Wikipedia (http://en.wikipedia.org/wiki/ISO/IEC_8859-11) says 
the following:

 >>>>
ISO-8859-11 is not a registered IANA charset name despite following the 
normal pattern for IANA charsets based on the ISO 8859 series. However, 
the close equivalent TIS-620 (which lacks the non-breaking space) is 
registered with IANA, and can without problems be used for ISO/IEC 
8859-11, since the no-break space has a code which was unallocated in 
TIS-620.
 >>>>

I would like to get your feedback on the following alternative proposals:

1) Leave everything as is.


2) Add an alias "ISO-8859-11" to the TIS-620 entry (acknowledging 
current practice and ignoring the official difference at 0xA0 (*)).


3) Add a new entry of the form:

Name: ISO-8859-11 (preferred MIME name)
MIBenum: [TBD]
Source: ISO/IEC 8859-11:2001
Alias: csISOLatinThai


I'm currently inclined to go with 2).
TIS 620-25

Re: Encoding Standard (mostly complete)

Leif Halvard Silli — 2012-04-26T13:24:08

Hi Anne!

First, another proposal: In the name/label table, state that "unicode" 
is another label for "utf-16" and that "unicodeFFFE" is another label 
for "utf-16be". 

Paradoxically, at least for HTML, then this is perhaps more relevant as 
a label for UTF-8 encodings than as a label for for UTF-16 encodings - 
however, this is also the case of for the other names of the UTF-16 
encodings.

Anne van Kesteren, Thu, 26 Apr 2012 13:44:05 +0200:


Roughly this was what I had in mind, starting with section 7:

  7 The <ins>standard</ins> encoding
  7.1 utf-8
  8 The legacy encodings
  8.1 The Single-byte legacy encodings
  8.2 The Multi-byte legacy encodings
  8.2.1 The Chinese (simplified) legacy encodings
  8.2.1.1 gbk
  8.2.1.2 gb18030
  8.2.1.3 hz-gb-2312
  8.2.2 The Chinese (traditional) legacy encodings
  8.2.2.1 big5
  8.2.3 The Japanese legacy encodings
  8.2.3.1 euc-jp
  8.2.3.2 iso-2022-jp
  8.2.3.3 shift_jis
  8.2.4 The Korean legacy encodings
  8.2.4.1 euc-kr
  8.2.4.2 iso-2022-kr
  8.2.5 The utf-1

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-26T11:44:05

On Mon, 23 Apr 2012 13:30:04 +0200, Leif Halvard Silli  
<xn--mlform-iua< at >målform.no> wrote:

How would you title the subsections? I could not think of something good.



Done.



I think this should become clearer once this is integrated into other  
specifications. I rather not mention too much format specific things here.

RE: Encoding Standard (mostly complete)

Shawn Steele — 2012-04-23T18:24:20


As noted, FYI, IE is likely not to change, particularly if that'd require supporting code pages or variations that windows doesn't know about.

-Shawn

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-23T12:05:10

On Mon, 23 Apr 2012 13:57:03 +0200, Julian Reschke <julian.reschke< at >gmx.de>  
wrote:

Private Use Area of Unicode. In some browsers certain byte sequences in  
legacy encodings will yield PUA code points (rather than an assigned code  
point or U+FFFD). Combined with certain fonts they can end up as a  
"meaningful glyph".

Thus far only  
http://dvcs.w3.org/hg/encoding/raw-file/tip/index-macintosh.txt contains  
<Private Use> (Apple's logo), but it has been suggested for CJK indexes  
(and indeed typically Gecko/WebKit/Trident have such extensions).

Re: Encoding Standard (mostly complete)

Julian Reschke — 2012-04-23T11:57:03

Ack.

PUA?

Re: Encoding Standard (mostly complete)

Leif Halvard Silli — 2012-04-23T11:30:04

Anne van Kesteren, Tue, 17 Apr 2012 11:38:47 +0200:


(1) Just an idea - take it or leave it: Section 7 is called "The 
encoding" and that it only describes a single encoding - UTF-8. In 
order to emphasize "UTF-8" as _the_ encoding, how about collapsing 
section 8 to 13 into a single section named "Legacy encodings", with 6 
sub-sections?

(2) I would suggest that you, the first time you talk about "byte order 
mark", also introduce the abbreviation - "BOM". Currently, BOM occurs 
in section 13 while "byte order mark" occurs in section 6.

(3) Regarding the note "the byte order mark is considered more 
authoritative than anything else", then I would suggest specifying what 
"anything else" means. I suppose that it includes - or at least ought 
to include 

HTTP, 
<meta charset>, 
<meta http-equiv=Content-Type>, 
<?xml version="1.0" encoding="<anyvalue>" ?>
Manual encoding overriding by the user
    The above is valid for both XML and HTML.

   Unless this is listed/described, then I think one starts to

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-23T10:07:11

On Mon, 23 Apr 2012 11:06:12 +0200, Julian Reschke <julian.reschke< at >gmx.de>  
wrote:

That depends on the market. E.g. in Taiwan they still seem to be.



The Encoding Standard is not codifying Internet Explorer at the moment.  
It's a balance between compatibility with deployed content, existing  
browsers, avoiding PUA, and XSS concerns. (We might have to cave on  
avoiding PUA, we'll see.)

Re: Encoding Standard (mostly complete)

Julian Reschke — 2012-04-23T09:06:12

But then, Microsoft isn't the market leader anymore, right?

Given the combined market share of Chrome and Firefox, maybe it's not 
needed anymore to do exactly the same?

Best regards, Julian

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-23T07:10:52

In brief, the goal is to document the "web platform". So that existing  
implementations and new implementations have no trouble figuring out what  
to implement. In part that requires figuring out what existing content  
relies upon, which can determined to some extent by figuring out what the  
market leader is doing.

Re: Encoding Standard (mostly complete)

Doug Ewell — 2012-04-21T21:57:07

Masatoshi Kimura replied to Shawn Steele:


It looks like this was a bug fix to correct bad error handling of 
invalid data that could enable a cross-site scripting attack. Is this 
undesirable?


MS10-090 seems to be about auto-detection, not encoding behavior. 
Auto-detection becomes relevant when pages are not properly tagged.


Is that the goal here—to help browser implementers compete with 
Microsoft?

--
Doug Ewell | Thornton, Colorado, USA
http://www.ewellic.org | < at >DougEwell

Re: Encoding Standard (mostly complete)

Masatoshi Kimura — 2012-04-21T15:22:08

Actually MS11-057 changed the IE decoder [1].

Indeed, MS10-090 broke some ISO-2022-JP encoded Web pages [2]. Please
make your action consitent with your words.

Even if we ignored non-Unicode encodings, we will need a documentation
about UTF-16 anyway due to IE quirks (BOM overrides everything, default
endian is little-endian contrary to the Unicode Standard, etc).
Otherwise other browser implementors will have to (and did)
reverse-engineer to develop a competing browser. Obviously TUS is
useless about IE quirks.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=690225#c14
[2] http://support.microsoft.com/?kbid=2467659

RE: Encoding Standard (mostly complete)

Shawn Steele — 2012-04-20T18:05:55

Exactly.  Opera isn't the only one that gets those bug reports.  I get bug reports too.

User A visits web site W that was written to match JIS's latest and greatest EXACTLY.  That fails on IE because we don't know everything about JIS.
User B visits web site X that was written based on an older Linux interpretation.  That fails on IE because the Linux version differed from ours.
User C visits web site Y that wasn't tagged perfectly with an exact variation, but it succeeds on IE because it happens to be our behavior.
User D visits web site Z that is tagged perfectly, but it fails in IE because we don't recognize the name.  The developer didn't bother with IE so didn't realize the problem.

These are all mutually exclusive.  If I "fix" user A, then I risk breaking the rest, etc.  If I break user C & web site Y, they get really mad because they used to work, even though it wasn't "right."  The only way to resolve this is to update all (or at least many) of the documents (which will never happen).  The

RE: Encoding Standard (mostly complete)

Shawn Steele — 2012-04-20T17:36:39

Unfortunately this document doesn't "own" any of the other standards that it's summarizing.  As an software developer I'd rather "go to the source" than an intermediary document that may have inadvertently introduced a discrepancy from the actual authoritative standards.

I'm not suggesting that a single source wouldn't be nice, but rather that it's impractical.  Unless you can get Unicode to cede the definition of UTF-8 to your document, and the same with all the other standards, they're bound to be inconsistent or diverge.

It may be better as a pointer to the other standards, &/or documentation of quirks where other standards have been implemented differently and the pros & cons of those standards.

-Shawn

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-20T06:47:54

On Thu, 19 Apr 2012 20:00:53 +0200, Shawn Steele  
<Shawn.Steele< at >microsoft.com> wrote:

You keep saying in this context of me trying to explain why a browser  
handling *legacy* pages has a hard time knowing what to implement. It is  
starting to get annoying. If those pages used Unicode we would not  
continue to get bug reports.



Yes, that is why we perform content analysis to figure out what the best  
way to decode data would be. See e.g.  
http://lists.w3.org/Archives/Public/public-html-ig-zh/2012Apr/



The assumption is that neither of those is going to happen for data we  
still want to read in say a hundred years time.



This effort is not aimed at content authors.

Speaking of which, I've been a tireless advocate of utf-8 since before I  
knew how it worked. I wrote e.g.

http://annevankesteren.nl/2004/06/utf-8
http://annevankesteren.nl/2009/09/utf-8-reasons

And last night while you wrote your email I presented on the topic at a  
local developer meetup:

http://annevankesteren.nl/presentations

Re: Encoding Standard (mostly complete)

Anne van Kesteren — 2012-04-20T06:28:44

I think having a single specification to address all encoding questions is  
useful. It presents encoding algorithms in a consistent style and gives  
other specifications an simple reference.


The Unicode standard

* Does not address labels (e.g. I cannot find "utf8" in the PDF)
* Deals with byte order mark and utf-16 in a manner that is not matched by  
implementations
* Does not go far enough in defining error handling
* Is a PDF, which is really annoying on the web

RE: Encoding Standard (mostly complete)

Shawn Steele — 2012-04-19T18:00:53

Our implementation of encodings WILL NOT change.  Ever.  I also don't have the resources to validate that your standard matches our behavior.

I'm not at all trying to say that our implementations are "perfect".  (On the contrary, I've blogged quite often about how there are lots of variations in the wild.  We even have slight differences in our various SDKs.)  However, there are millions of our customers that depend on our current behavior.  If that behavior changes even slightly, then that will "corrupt" their data.

Our handling of shift_jis is probably the most severe of those, where the standard has evolved beyond what we support, but we can't change without breaking people.  


Use Unicode.  


Use Unicode.  Even if you figure out exactly what every browser is doing, you still have no idea what browser/version the page was targeting.  Even if you created a perfect version of the ABC encoding (placeholder for your favorite encoding), and convinced all of the browsers to adopt the perfect ABC

RE: Encoding Standard (mostly complete)

Doug Ewell — 2012-04-19T17:58:57

Copying the Unicode mailing list.

Masatoshi Kimura <VYV03354 at nifty dot ne dot jp> wrote:


I remembered reading a statement from UTC that interpretation of an ill-
formed sequence was supposed to terminate as soon as the sequence was
determined to be ill-formed. Conformance definition C10 does say:


A lead byte of 11111000₂ is ill-formed.

And in fact, the section of TUS that Masatoshi quoted goes on to say:


However, this description does use the word "should," not "must," and it
goes on (on the same page) to offer a table with three "possible
alternative approaches" for mapping an ill-formed UTF-8 sequence into
characters. It recommends the method described above, but allows the
other two.

So the bottom line is that Masatoshi is right: the Unicode Standard does
not specify that a decoder *must* respond to an invalid lead byte as I
said, only that it *should*. I agree that this is unnecessarily vague.

Whether this calls for a complete recasting of the definition of UTF-8
by WHATWG, or by any indiv

Re: Encoding Standard (mostly complete)

Masatoshi Kimura — 2012-04-19T16:48:20

Where TUS defines this? It seems to contradict TUS 6.1.0 p.96:
http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf#page=42
|Although a UTF-8 conversion process is required to never consume
|well-formed subsequences as part of its error handling for ill-formed
|subsequences, such a process is not otherwise constrained in how it
|deals with any ill-formed subsequence itself. An ill-formed subsequence
|consisting of more than one code unit could be treated as a single
|error or as multiple errors. For example, in processing the UTF-8 code
|unit sequence <F0 80 80 41>, the only formal requirement mandated by
|Unicode conformance for a converter is that the <41> be processed and
|correctly interpreted as <U+0041>. The converter could return
|<U+FFFD, U+0041>, handling <F0 80 80> as a single error, or
|<U+FFFD, U+FFFD, U+FFFD, U+0041>, handling each byte of <F0 80 80> as a
|separate error, or could take other approaches to signalling <F0 80 80>
|as an ill-formed code unit subsequence.
It is exactly a purpose of