gmane.ietf.http-wg

Re: Design Issue: GZIP flag on DATA Frames

Roberto Peon — 2013-05-22T00:27:44

I don't see why it is a one or the other choice.

lot of implementations, or went through a proxy which stripped it.

Requiring that it be supported enabled servers to send the data compressed.
This was essentially the result of years of being frustrated with not
knowing when it was safe to at least be able to send with a baseline
compression.

Unfortunately, gzip isn't the best compressor in the world, so we will
always need since mechanism for negotiating a compression scheme.
Http's scheme works, and many of its flaws are unavoidable in any system
with proxies.

Having played with compression at the protocol layer, my experience is that
it adds complexity where it is often not warranted and it results in (at
best )awkward apis.

-=R
If that's the case, and it's better to just handle data compression at
the higher levels, then the requirement that gzip MUST always be
supported independent of what is specified by the accept-* header
ought to be removed and implementations ought to just continue letting
the application decide which mechanism is best.

My goal here is to simplify things wherever possible. If we can
deprecate (or greatly simplify) accept-/transfer-/content-encoding at
the semantic layer and just say that the framing layer will handle
compression, then that's one less optional bit of complexity that us
application-level developers have to deal with.

On Tue, May 21, 2013 at 2:36 PM, Roberto Peon <grmocg-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:
my
writing
HTTP
work
http://bjoern.hoehrmann.de
http://www.bjoernsworld.de

Re: notes on http2 draft

Patrick McManus — 2013-05-22T00:16:00

Can't wait for the summary notes :)



sgtm

Re: Design Issue: GZIP flag on DATA Frames

Patrick McManus — 2013-05-21T23:34:48

James, I think the existing spec language is fine. Its there based on
implementation experience (which has been collected in scenarios with no
implicit accept-encoding, as well as early spdy versions that applied gzip
on a spdy frame basis, along with incantation in there now which has worked
well imo) - you're sort of frenetically re-enacting the entire history of
the feature's many changes all in one day.

The way it is, the skids are greased for deflate as a LCD and the door is
open in the usual way to other approaches.


On Tue, May 21, 2013 at 6:07 PM, James M Snell <jasnell-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Design Issue: GZIP flag on DATA Frames

Frédéric Kayser — 2013-05-21T23:15:28

Hello,
HTML5 accepts only UTF-8 encoding, there are better ways than Deflate to compress Unicode texts, bzip2 to start with. Deflate has no clue about UTF-8 since it is byte oriented, its search window is limited to 32 kilo bytes (in UTF-8 a single Devanagari character –used for Hindi and other languages in India– takes 3 bytes which seriously reduces the size of text that can actually be used as a reference for string matching, the same goes for other scripts like Cyrillic (Russian, Ukrainian, Bulgarian…), Greek,  Hebrew, Arabic… since they can no longer rely on single byte charsets and UTF-8 means 2 bytes per character for those).

For web performance having a compression scheme that could recognize and reverse/redo base64 encoding (Data URI, RFC2397) to handle "binary blobs" inside text files would be very appreciated.

Deflate misses some flexibility since it has no super fast mode à la LZ4 that would still provide decent compression but at much lower CPU cost (no entropy coding), nor something heavier on the other side (LZMA like).

Deflate was a nice compression scheme in the 90s, but the World (Wide Web) has changed since the 90s, look how archivers handle text files nowadays: they switch to PPMd, bzip2… because Deflate is outdated.

Compressing the headers is a good idea, but thinking about new compression schemes for the payload should not be overlooked.

Regards
Frédéric Kayser

Le 21 mai 2013 à 19:17, Poul-Henning Kamp a écrit :

Re: Design Issue: GZIP flag on DATA Frames

James M Snell — 2013-05-21T22:07:32

If that's the case, and it's better to just handle data compression at
the higher levels, then the requirement that gzip MUST always be
supported independent of what is specified by the accept-* header
ought to be removed and implementations ought to just continue letting
the application decide which mechanism is best.

My goal here is to simplify things wherever possible. If we can
deprecate (or greatly simplify) accept-/transfer-/content-encoding at
the semantic layer and just say that the framing layer will handle
compression, then that's one less optional bit of complexity that us
application-level developers have to deal with.

On Tue, May 21, 2013 at 2:36 PM, Roberto Peon <grmocg-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Design Issue: Separate HEADERS and PRIORITY Frames, Eliminate HEADERS+PRIORITY

James M Snell — 2013-05-21T21:55:53

Sorry, I wasn't clear in my initial response. Yes, there is some state
that would need to be allocated but not the same as that when the
initial HEADERS frame is received, for instance.


Not particularly, because we'd still have the question of when to use
HEADERS+PRIORITY vs. the combination of HEADERS and a CHANGE_PRIORITY.
Can HEADERS+PRIORITY be used any time? For instance, could I send an
initial HEADERS frame on a stream then later send a HEADERS+PRIORITY
on the same stream? I honestly don't care how it ultimately ends up so
long as (a) It's the simplest thing that could possibly work and (b)
Is easy to explain in the spec and easy for a developer to implement.


I would very much like to see us mandate that streams always initiate
with a HEADERS / HEADERS+PRIORITY frame.

- James

Re: Design Issue: GZIP flag on DATA Frames

Poul-Henning Kamp — 2013-05-21T21:53:36

In message <8ipnp8hapu1949rd0mahtoasm60mm1i46v-cYIUb816AqSJigVVbuHhyrWtS95+f1mF< at >public.gmane.org>, Bjoer
n Hoehrmann writes:


Ok, so let me clarify:

I have always hated the confusion in HTTP/1 between content compressed
for transport purposes, and content compressed for application purposes.

In the former class you have HTML, javascript, CSS etc.

In the latter class you have things like .tgz, gzip'ed XML etc.

It's the former class I want to cut all the way into the bone
and take out of "Accept-Encoding" negotiation.

Accept-Encoding should be reserved only for the second class.

Re: notes on http2 draft

Martin Thomson — 2013-05-21T21:49:06

HTTP/2.0 originally had exactly this feature and it was removed.  It
was removed from SPDY shortly after the two specifications diverged.

https://github.com/http2/http2-spec/issues/46

It turns out that a compression bit is a) not especially useful
without knowledge of what the payload is going to be, and b)
confusing.

Re: notes on http2 draft

Martin Thomson — 2013-05-21T21:45:48

Thanks.  Fixed.


Scrubbing done.  Turns out that "message" was misused almost
everywhere it appeared :)


I don't know how to do that, except by trimming "BAR" down to "BA" (or
"GO" or some other shed-tone).  That might need to be tested.


I think that we plan to discuss this at the interim.  I don't plan to
take a position on this, except that whatever we choose needs to be
consistently applied and motivated, so maybe I can do a (poor) Patrick
impersonation for you.


I would be perfectly happy with this position, if only because a) it
resolves the issue, and b) ends up with less text in the draft.


This is an interesting one.  The view that I think that James has
taken is that the compression context is evaluated at a lower layer of
the logical stack than streaming.  That would allow this to remain
true, at least superficially.

I get the point though.  Do you think that qualifying the statement to
say that streams are subject to independent flow control and
life-cycles would be enough?


I think that the intent is to avoid precluding any chance of
multiplexing websockets in here.  HEADERS+PRIORITY might be used
there, but it might not be necessary.


PUSH_PROMISE doesn't create a stream.  It promises to.  This is
important because it allows a pushing server to promise more streams
that the maximum stream concurrency limit set by the client.  Hence
the caveat.


We haven't had a proposal for a change-priority frame yet.  I'm not
sure that H+P is going to be optimal for this case.


No particular reason.  Other than it being easier.  Switching to


Righto.  MUST.


Yes.  Express > implied.


I think that this was marked as so when the COMPRESSED bit was
removed.  I don't think that this proscription on the upper layer is
unreasonable, even if it is a bit of a layering violation that this
specification doesn't strictly need to address.

I've moved this to the response processing side and clarified the
statement.  If there are objections to this, the paragraph is easy
enough to remove.


Gotcha, and 4.2.3 fixed (to the converse) too.  Servers have no real
business setting priority (for HTTP).

Re: Design Issue: Separate HEADERS and PRIORITY Frames, Eliminate HEADERS+PRIORITY

Roberto Peon — 2013-05-21T21:43:54

Sending the PRIORITY frame *MUST* cause state allocation at the receiver,
else it was useless to send before the HEADERS frame. As you point out, at
minimum it must allocate a stream ID and priority field, and for most
implementations it will also need to include so mechanism of pointing out
that the headers don't exist, so, probably between 16 to 24 bytes worth of
allocation on a 64 bit machine.

If the PRIORITY frame was renamed to CHANGE_PRIORITY, would that clarify
anything? Priority changing is the current intent of that frame type.

btw, I am not particularly partial to the "any frame opening up a stream"
thing. I'm not completely against it though :)
My reason for slightly preferring that streams must begin with HEADERS or
HEADERS+PRIORITY is that it is an explicit statement of intent, and thus
off-by-one, uninitialized var, etc. errors are more likely to be detectable
in a world where such is required.



-=R


On Tue, May 21, 2013 at 2:19 PM, James M Snell <jasnell-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Design Issue: GZIP flag on DATA Frames

Roberto Peon — 2013-05-21T21:36:39

The proposal does force anything other than gzip to use an alternate code
path for parsing/interpretation, which does not inspire confidence given my
experience with backup-generator problems, and it probably involves writing
*more* code today than the alternative of leaving it where it is in the
HTTP semantic layer.
-=R


On Tue, May 21, 2013 at 2:24 PM, James M Snell <jasnell-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Design Issue: GZIP flag on DATA Frames

Bjoern Hoehrmann — 2013-05-21T21:32:48

Having a "gzip flag" as, in effect, a shorthand for using those headers
would make it very clear how to deploy alternative compression schemes,
and would indeed address my concerns. I understood Poul-Henning Kamp to
be considering going one step further and offering no means to define
new compression schemes.

Re: Design Issue: GZIP flag on DATA Frames

James M Snell — 2013-05-21T21:24:04

This does not preclude the use of alternative compression schemes. If
someone chooses, it would be possible, for instance, to continue using
accept-/content-/transfer-encoding at the http semantic layer and
simply not set the GZIP flag on the DATA frame. Having the GZIP flag
would just provide an approach that would make that unnecessary in the
most common cases today. If, at some point in the future a new, more
efficient better compression algorithm overtakes gzip as the
predominant mechanism, the protocol can be updated to reflect that
fact (i.e. with a new protocol version string, etc).

On Tue, May 21, 2013 at 2:17 PM, Bjoern Hoehrmann <derhoermi-hi6Y0CQ0nG0< at >public.gmane.org> wrote:

Re: Design Issue: Separate HEADERS and PRIORITY Frames, Eliminate HEADERS+PRIORITY

James M Snell — 2013-05-21T21:19:38

On Tue, May 21, 2013 at 10:30 AM, William Chan (陈智昌)
<willchan-F7+t8E8rja9g9hUCZPvPmw< at >public.gmane.org> wrote:

There is no reason to assume that sending a PRIORITY frame first would
trigger stream state allocation at the receiver. At most, it would
reserve the stream ID and store the priority value. The full state
allocation would not occur until the HEADERS frame is received. That
said, I'm not 100% dead set on removing HEADERS+PRIORITY, I would just
like to simplify the protocol where it makes sense to, and even then
only after it's been proven out in implementation. Having separate
HEADERS, HEADERS+PRIORITY and PRIORITY frames is confusing, if we can
do without separating them, we ought to do so.

- James

Re: Design Issue: GZIP flag on DATA Frames

Bjoern Hoehrmann — 2013-05-21T21:17:28

I think people will create an even bigger mess if they're forced to work
around lack of support for alternative compression schemes in HTTP/2.0.

Re: Design Issue: GZIP flag on DATA Frames

Poul-Henning Kamp — 2013-05-21T20:52:53

In message <fnlnp8t14lal0uk5suouc2uuk3uk4429dt-cYIUb816AqSJigVVbuHhyrWtS95+f1mF< at >public.gmane.org>, Bjoer
n Hoehrmann writes:



You forgot half the "worth-while" part:  The compatibility issues.

Even something as trivial as "deflate" vs. "gzip" was too hard for
some people to get right.

Re: Design Issue: GZIP flag on DATA Frames

Bjoern Hoehrmann — 2013-05-21T20:45:43

There are many existing compression schemes that considerably outperform
Deflate in the compression ratio, some even at comparable consumption in
resources needed for decompression and, less so, compression. Especially
for something like "large JavaScript library" that you would compress
only once, something like http://en.wikipedia.org/wiki/.xz is quite near
a point where I would expect people to call for adoption of a new scheme
(with a filter optimized for such content > 10% better compression seems
very plausible to me). If the protocol cannot support adoption of a new
scheme, there would be pressure to move compression onto a lower level,
think "application/compressed-javascript", and I would rather avoid go-
ing there. My http://bjoern.hoehrmann.de/pngwolf/ was able to strip se-
veral kilobytes off the Google homepage, that already seemed quite worth
it...

Re: Design Issue: GZIP flag on DATA Frames

Bjoern Hoehrmann — 2013-05-21T20:19:51

In the Hybi Working Group many participants insisted on being able to
serve very large frames using `sendfile(...)`-style APIs. Deflate re-
quires sending additional data every couple of kilobytes, which would
interfere with that. I haven't studied the actual impact or how such
an argument applies to the most recent HTTP/2.0 proposals, and suspect
it's actually not a terribly big issue, but I would expect objections;
accordingly, such a proposal would have to come with some data that
alleviates concerns in this area.

Re: Design Issue: GZIP flag on DATA Frames

Martin Nilsson — 2013-05-21T19:17:42

On Tue, 21 May 2013 18:25:37 +0200, Poul-Henning Kamp <phk-HF+kCXwiSi60G1uiEUka0Q< at >public.gmane.org>  
wrote:


The deflate format as defined in RFC 1951 requires 5 bytes added  
(injected) per 65k data. Also the two formats actually used in HTTP  
(deflate, RFC 1950, and gzip, RFC 1952) requires cheksums of the entire  
data added, so it's not like uncompressed mode is free.

/Martin Nilsson

Re: Design Issue: Separate HEADERS and PRIORITY Frames, Eliminate HEADERS+PRIORITY

William Chan (陈智昌 — 2013-05-21T17:30:26

Thanks for describing these cases now. I had not thought of them.

If everyone's sold on reprioritization, then let's go ahead and do this and
have the debate about ditching HEADERS+PRIORITY or not. I want to keep it.
I don't like the idea of sending a PRIORITY frame first. Is sending a
PRIORITY frame going to trigger stream state allocation at the receiver?
What's the expectation? And if you don't have a priority for the HEADERS,
then you have the race that Roberto described.


On Tue, May 21, 2013 at 2:09 PM, Patrick McManus <pmcmanus-4eJtQOnFJqFBDgjK7y7TUQ< at >public.gmane.org>wrote:

Re: Design Issue: GZIP flag on DATA Frames

Poul-Henning Kamp — 2013-05-21T17:17:43


And what is the plausibility that any new compression schemes will ever
make that worth-while ?

It's not nill, but it makes a convincing impression of nill.