gmane.ietf.http-wg

http2 draft feedback on flow control

Patrick McManus — 2013-05-22T19:32:05

Hello,

While reading the flow control text again there was some ambiguity in my
mind about whether or not the SETTINGS_INITIAL_WINDOW_SIZE adjusted the
connection window or just all current and future stream windows. I've
decided the text is really going for "just stream windows - connection
windows are only adjusted by WINDOW_UPDATE with stream=0" (an
interpretation I favor)..

I think it would be nice if it said that explicitly to clear up confusion.
There are some PoC implementations of this floating around out on them thar
Internets (admittedly they may predate the current spec language) and I've
seen different interpretations of what is logical.. so it makes sense to be
explicit.

A couple related followup tweaks to the text:


section 3.8.9.2

* *

*"When a HTTP/2.0 connection is first established, new streams are created
with an initial flow control window size of 65535 bytes. The connection
flow control window is 65536 bytes. Both endpoints can adjust the initial
window size for new streams by including a value for
SETTINGS_INITIAL_WINDOW_SIZE in the SETTINGS frame that forms part of the
connection header. *"

I suggest s/in the SETTINGS frame that forms part of the connection
header/with a SETTINGS frame/ .. The section is titled "Initial Flow
Control Window Size" so the existing text could be ok if we mean initial to
be at connection establishment time, but I think it more clearly applies to
the life cycle of each stream (at least when talking about stream windows).

**
**

**


*When the value of SETTINGS_INITIAL_WINDOW_SIZE changes, a receiver MUST
adjust the size of all flow control windows that it maintains by the
difference between the new value and the old value. *

uh-Oh.. "all flow control windows that it maintains" definitely includes
the connection window :( I suggest s/flow control windows/stream flow
control windows/
-Patrick

Design Issue: GZIP flag on DATA Frames

James M Snell — 2013-05-21T16:21:27

https://github.com/http2/http2-spec/issues/100

Currently the spec includes a requirement that all user-agents MUST
support gzip.. specifically:

  User-agents MUST support gzip compression.
  Regardless of the Accept-Encoding sent by
  the user-agent, the server may always send
  content encoded with gzip or deflate encoding.

If we're going to include this requirement, it makes more sense to do
this at the framing layer rather than the HTTP semantic layer. We can
do so easily by defining a GZIP flag on the DATA frame type. If set,
the payload of the DATA frame is compressed.

Doing so largely eliminates the need for the
accept-/transfer-/content-encoding mechanisms at the http semantic
layer.

Design Issue: Separate HEADERS and PRIORITY Frames, Eliminate HEADERS+PRIORITY

James M Snell — 2013-05-21T16:10:50

https://github.com/http2/http2-spec/issues/99

With regards to the discussion over stream re-prioritization, I suggest:

1. Drop the HEADERS+PRIORITY frame type.
2. Create a new separate PRIORITY frame type whose payload is the
Priority value, no frame-specific flags.
3. The PRIORITY frame becomes the only way to set/change the priority
for a stream.

If it is necessary to allow an endpoint to establish the priority of
stream prior to actually initiating the stream, we can allow sending a
PRIORITY frame before the initial HEADERS frame. Doing so would
effectively reserve the stream id (in the same general manner
PUSH_PROMISE does).

The advantages of this approach are:

1. It eliminates any possible confusion and complexity about when to
use HEADERS+PRIORITY vs. HEADERS
2. It provides a single way of setting/change stream priority (as
opposed to using HEADERS+PRIORITY plus a separate CHANGE-PRIORITY
frame)

notes on http2 draft

Patrick McManus — 2013-05-21T15:01:53

Hi All - While I've certainly closely followed the deltas to the http/2
draft I haven't actually sat down and read the thing end to end in a while.
As I have a conflict and won't be able to join you all in CA in June, I sat
down today  to do a full read through and have some feedback on the current
state of the draft. There is a mixture of pure editorial here as well as
some more subtstantive opinion stuff.

First, I want to say thanks to the editors - overall this document is great
progress in terms of readbility and organization.
*
*
*
*
*Section 1.
*
*Furthermore, HTTP/1.1 header fields are often repetitive and verbose,
which, in addition to generating more or larger network packets, can cause
the small initial TCP congestion window to quickly. This can result in
excessive latency when multiple requests are made on a new TCP connection.*

[editorial]
Furthermore, HTTP/1.1 header fields are often repetitive and verbose,
which, in addition to generating more or larger network packets, can cause
the small initial TCP congestion window to quickly fill. This can result in
excessive latency when multiple requests are made on a single new TCP
connection.


1.2
*
message:**A complete sequence of frames.*

I'm not sure that does a lot for me as a definition. It should at least say
something about them sharing the same stream ID.. or am I describing a
stream? Is a message a unidirectional stream? as I said.. it doesn't do a
lot as a definition. The document refers to "window update messages" and
"goaway messages" but I think it means frames in those cases.. it also
talks about "receiver of a message" sending WINDOW_UPDATE which makes it
sound like a message is any data frame (not the complete sequence of
them)... and then again we also talk about "HTTP Messages" which are
something distinct..

I suggest we scrub the term message from the document (and this section)
except when it refers to HTTP messages.


3.2 Connection Magic

Can we make that bikeshed a multiple of 4 bytes long to at least
conveniently align the SETTINGS frame that follows?

3.3.1

* Implementations MUST ignore unsupported and unrecognized frame types. *

I think invalid frame types should be session errors of the MUST NOT send
variety. I know the list has been talking about this lately and I haven't
had an opportunity to chime in. Be liberal in what you receive is overrated
and often fraught with security problems - we can rev the protocol with
ALPN at Internet scale in order to sanely guide extensions as needed. I
know design committees love open ended extensions so my view won't pervail,
but this is exactly the kind of thing that leads to interop doom.

3.3.2

*Implementations with limited resources might not be capable of processing
large frame sizes. Such implementations MAY choose to place additional
limits on the maximum frame size. However, all implementations MUST be
capable of receiving and processing frames containing at least 8192 octets
of data.
*
I'm pretty confident we are just inventing complexity here for no good
reason. The tiny universe of implementations that can cope with 8KB but not
64KB is not worth the complexity. One of the advantages of going with a 16
bit frame size is that it should be small enough for everything to handle
it.

If there is reason to believe 64KB is too big for a population large enough
to care about (remembering of course that HTTP/2 is not a ticket to
Internet admission - if you're really so small that you can't read 64KB
frames the muxing of HTTP/2 is going to be a real challenge too; you're
probably best left using a different protocol. Just coap :)), then let's
lower the max frame size instead of reinventing Path MTU Discovery
complexity. But I think we can just drop this section and require everyone
to deal with 64KB.

3.4

*A "stream" is an independent, bi-directional sequence of frames*

Due to the (expected) compression requirements the frames aren't really
independent of other streams. I know this mistake comes up pretty commonly
on spdy-dev from new implementers of that protocol, so it probably helps to
avoid saying independent here. (section 5.3 does it too).

3.4.1

*Rather, new streams are established by sending a frame whose stream
identifier field references a previously unused stream identifier. *

That's a little too loose. Streams are created by the client through
HEADERS+PRIORITY (4.2.2) and by the server through PUSH_PROMISE. The text
as-is makes it sound more free flowing than that.

*The identifier of a newly established stream MUST be numerically greater
than all previously established streams from that endpoint within the
HTTP/2.0 connection, unless the identifier has been reserved using a
PUSH_PROMISE (Section 3.8.5<http://http2.github.io/http2-spec/#PUSH_PROMISE>)
frame.*

Likewise, PUSH_PROMISE really creates the stream (4.3.1) from the server..
I don't see a reason for the caveat here.


3.4.2

I thought one of the takeaways at Tokyo was to define a change-priority
frame.. Can we do that now? is the intention to use a H+P without any
headers at any point in the stream? If so, I think that should be called
out so that server implementation's don't freak out at seeing H+P at
strange points in the stream.. and I think there is some language in 4.2.2
that could be interpreted as meaning certain colon headers are required to
be in every H+P


3.8.5

*The PUSH_PROMISE frame (type=0x5) is used to notify the peer endpoint in
advance of streams the sender intends to initiate.*

I think I must have missed something on list about this. Why is the
language here general (peer, sender).. why define this so that clients can
generate PUSH_PROMISE ?

3.8.9.4

*After a receiver reads in a frame that marks the end of a stream (for
example, a data stream with a FINAL flag set), it ceases transmission of
WINDOW_UPDATE frames. A sender is not required to maintain the available
flow control window for streams that it is no longer sending on. *

First - since this is a spec, that should be "it MAY/SHOULD/MUST cease
transmission of WINDOW_UPDATE frames". I suggest MUST.

Second - The language isn't clear if this applies to session windows or
just the stream window. I think it is just the stream window and the
receiver still needs to ack^H^H^HWINDOW_UPDATE the session window after the
FINAL flag, but we should say explicitly.

4.2.2

*User-agents MUST support gzip compression. Regardless of the
Accept-Encoding sent by the user-agent, the server may always send content
encoded with gzip or deflate encoding.
[rfc.comment.11<http://http2.github.io/http2-spec/#rfc.comment.11>:
Still valid?]*

I support that still being valid. This is an important performance property
to rely on.

also 4.2.2

*The client initiates a request by sending a HEADERS+PRIORITY frame [...]
if the server receives a data frame prior to a HEADERS or HEADERS+PRIORITY
frame
*

I think that means just "prior to a HEADERS+PRIORITY frame" which is what
is required to open the stream...
4.2.3

*The server responds to a client request using the same stream identifier
that was used by the request. An HTTP response begins with a HEADERS frame.
[...] If the client receives a data frame prior to a HEADERS or
HEADERS+PRIORITY

*
Again, I think that means "prior to a HEADERS" [NOT "or H+P"]. Can server's
send H+P frames? If so, why? and if there is a compelling why, why can't
the response begin with H+P?


hope that helps.

Preliminary agenda for the SF Interim

Mark Nottingham — 2013-05-20T03:54:34

As you can probably guess, we haven't been able to get implementations moving before this Interim, but I'd very much like to exit it being able to get there quickly.

With that in mind, I've asked the editors to publish a draft -03 by early next week, to give everyone enough time to review the draft.

Likewise, I've asked Herve and Roberto to get their header proposal(s) lined up and ready for discussion at the F2F.

Please come into the meeting having read ALL of those documents carefully (once updated), as well as the issues list <https://github.com/http2/http2-spec/issues>.

If there are particular issues that you believe are important to get sorted out for the first implementation draft, please bring them up on-list BEFORE the meeting, so that we can make that time as productive as possible. The current list of such issues is <https://github.com/http2/http2-spec/issues?milestone=1&page=1&state=open>

A _rough_ agenda for discussion:

* Administrivia
   - Blue sheets
   - Introductions
   - Scribe Selection
   - Agenda Bashing
   - NOTE WELL
   - Ground rules for the meeting
* Document review [editors]
  - Open editors' questions
* Header compression
* Server push walkthrough
   * PUSH_PROMISE
* Flow Control overview
* Prioritisation review
* Upgrade / negotiation
* Other issue discussion
* First implementation planning

As always, suggestions appreciated; this will firm up over the next week.

Regards,

--
Mark Nottingham   http://www.mnot.net/

adding Header Continuation

Mark Nottingham — 2013-05-20T03:29:50

Roberto submitted a pull request for header continuation:
  https://github.com/http2/http2-spec/pull/98

Without going into the fine details of the text, are we ready to have such a thing?

My reading is that people are OK with this, at least to the point of putting it in the implementation draft (i.e., implementation experience may cause us to pull it back out again).

Speak up if you disagree.

Regards,


--
Mark Nottingham   http://www.mnot.net/

"Magic" string

Mark Nottingham — 2013-05-20T03:22:16

We've currently incorporated the magic as:


That string was based upon the quick testing I did a while back. 

I'm curious to know how people feel about this; while it's cosmetic, do we want to do something a little more... expected, like:

START * HTTP/2.0\r\n\r\nGO\r\n\r\n

?

Regards,

--
Mark Nottingham   http://www.mnot.net/

Part 2: Editorial changes involving "instance of time"

Peter Occil — 2013-05-15T01:35:47

As an editorial change I suggest changing Section 7.1.1 of Part 2 in the following ways: 

* In the phrase "a reasonable approximation of the current instance in Coordinated Universal Time", change "instance" to "instant".
* In the sentence "An HTTP-date value represents time as an instance of Coordinated Universal Time (UTC).", change "instance of" to "instant in".

--Peter

Implementation Notes on Server Push

Patrick McManus — 2013-05-14T19:14:59

I'm wrapping up a Firefox implementation of spdy/3 server push and wanted
to provide some ietf level feedback because that mechanism is largely
intact in the current http2 work in progress draft.


   - 304's seem to be a common cause of wasted pushed streams. I would see
   servers respond with a 304 for index.html and still push 200 responses for
   a.css and b.js when in a non-push world that would have been either 1 or 3
   304's. Maybe we should have a rule that you can only push to an
   assoc-stream of 2xx ?
   - There is language in the spec that if the client resets a stream that
   it implicitly resets any associated streams too. That was complicated to
   implement and pretty much broke my stream state model internally - while
   researching it it appears that mod_spdy and chrome don't implement it at
   all. The spec has an editorial note about removing that feature and I favor
   that removal.
   - I found flow control for pushed streams immensely helpful. It lets the
   client bound how much data can be pushed before there is a local GET
   matched up with that. Relatedly, I changed my default SETTINGS window size
   from a ~infinite value to be a smaller push-apropos value and then
   pipelined a window update with each odd SYN_STREAM to make pulled streams
   ~infinite again while preserving the smaller limit for push and this worked
   fine with all existing servers to my pleasant surprise. That seems to mean
   at least the spdy/3 windowing mechanism is simple enough for people to get
   right.
   - As with other parts of the spec, I was faced in some implementations
   with some very large data frames (90+KB) from servers when testing. Such
   frames are impervious to CANCEL or any mechanism we might use to change
   priorities driven by the client :(..  HTTP2 is doing the right thing by
   creating a smallish max frame size.


Obviously this is implementation experience with testing and isn't real
operational experience yet.. that will come.

-P

How to pronounce "charset"?

Yi, EungJun — 2013-05-13T13:00:17

Hi, all.

I am translating the book "HTTP: The Definitive Guide" from English to
Korean, and I feel difficult to translate the word "charset" to Korean. So
I have just decided to translate the word as it sounds.

But I don't even know how to pronounce the word correctly. Many people
pronounce "charset" as "character set" but I think that is incorrect
because the word does not mean "character set".

Can anyone tell me how to pronounce?

Thanks in advance.

http/2 test framework

William Chan (陈智昌 — 2013-05-12T21:56:32

I know we talked about this at the last interim meeting, but as far as I
know, nothing has happened on this front. Has anyone started on this yet?
I'd love to see us make progress on this front. I also think that this will
help clarify corners of the spec, since everytime we raise an ambiguity, we
really oughta write a test for it.

Here are my rough thoughts here:
* Again, host all code/configuration on github, probably in the http2
organization again
* Run a http2 test server in EC2. Host client test on it
* Have code for a server test. Uses a test client and a set of test pages.
Servers must host the pages at well defined paths.
* Um, think something up for proxies. Not hard, but I'm running outta time
here.

Mostly, I want to get people thinking about this and wanted to see who's
interested in working on this. I actually don't want to work on it since as
many people know, I'm technically part-time (20%) employed now
hacking/traveling. But if no one steps up to contribute here, I will
probably do so. That would be terrible because I'm busy traveling most of
the time, and if people left it up to me, I would write everything in Go
just to learn it. I know Mark has some basic SPDY code in Python that would
be a good foundation for this work.

So yeah, let's get started on this testing thing. I think it's important.

Design: Frame type extensibility issues

James M Snell — 2013-05-11T16:16:13

https://github.com/http2/http2-spec/issues/95

(btw, I'm posting these so that the issues are documented.. if there's
no time to discuss these issues now, or if it makes more sense to
discuss them later on in the process, that's obviously perfectly fine)

There are a couple of issues relating to frame type extensibility:

1. Are frames end-to-end or hop-by-hop? This is currently undefined.
Control frames appear to always be hop-by-hop, Header bearing frames
might or might not be hop-by-hop, Data frames appear to always be
end-to-end. What about new extension frame types?

2. Are new unsupported frame types subject to flow control? How can we
differentiate new control frame types vs. data frame types?

3. There is no prescribed mechanism for rolling out support for new
standard frame types vs. private use frame types.

Recommendation:

One approach we can take to dealing with this is to segment the 8-bit
frame type namespace into two distinct categories and reserve a
limited number of frame types from each for "private use". That is,

0xxxxxxx => Control Frames
1xxxxxxx => Data Frames

Control Frames (0x00-7F) are always hop-by-hop and are not subject to
flow control. Frame types 0x6B-7F would be reserved for "private use",
meaning that these types could not be registered within the IANA
registry.

Data Frames (0x80-FF) are always end-to-end and are subject to flow
control. Frame types 0xEB-FF would be reserved for "private use".

The frame type of the existing DATA frame would change from 0x00 to
0x80, all other existing frame types would remain unchanged.

This approach gives us a clear way of dealing with extension frames,
flow control, etc without introducing undue complexity.

Design: Ignored Unknown Frame Types and Intermediaries

James M Snell — 2013-05-11T15:27:40

In the current draft, endpoints are required to "ignore" unknown and
unsupported frame types. What's not yet clear, however, is whether
such frames are required to be forwarded on by intermediaries that do
not support them.

In other words, A talks to C via reverse proxy B. A sends a stream
that includes EXTENSION_FRAME_TYPE that is unknown to B. Is B...

A) Required to drop the frame silently without forwarding it on to C
B) Required to always forward the frame on to C
C) Neither, B can do whatever it wants

There is an obvious impact here on the future deployment of new
extension frame types. If the answer is A or C, we'll have to wait on
infrastructure support to use new frame types, which would be
unfortunate.

- James

Design Issue: Overlong Frames

James M Snell — 2013-05-09T17:26:16

In going through a number of issues relating to frame sizes, I note
that the spec currently does not deal with the issue of "overlong" or
padded frames. That is, what happens if a frame contains more data
than what is explicitly called for in it's definition.

For instance, the GOAWAY frame currently defines it's payload as a
32-bit error code. What happens if that frame contains more than
32-bits? .. e.g.

  00 40 07 00 00 00 00 00
  0A BC DE FF FF FF FF FF

An implementation that is not being careful could completely miss the
extra junk bytes here. For GOAWAY it's obviously not too much of a
concern, but the risk for abuse exists for all frames that define a
specific structure for the payload data.

Recommendation: Adding a short statement that a PROTOCOL_ERROR MUST be
returned if a frame contains more bytes than what is expressly
specified in the frame definition.

- James

3.5.1 Connection Error Handling

William Chan (陈智昌 — 2013-05-09T14:24:18

http://http2.github.io/http2-spec/#ConnectionErrorHandler has this section
which says:
"""
An endpoint that encounters a connection error MUST first send a GOAWAY
(Section 3.8.7) frame with the stream identifier of the last stream that it
successfully received from its peer. The GOAWAY frame includes an error
code that indicates why the connection is terminating. After sending the
GOAWAY frame, the endpoint MUST close the TCP connection.
"""

I'm not sure how to interpret the last line. I see two possible
interpretations:
1) At some point after sending the GOAWAY, it must close the connection.
2) It must close the connection immediately after sending the GOAWAY.

I think interpretation (1) is boring and useless. I think interpretation
(2) would probably be a spec error, since we want to allow time for
completing processing on accepted streams.

I think the description in 3.8.7 gets it mostly correct -
http://http2.github.io/http2-spec/#GOAWAY. Except of course for the noted
issue in the text.
"""
After sending a GOAWAY message, the sender can ignore frames for new
streams.

[rfc.comment.8: Issue: connection state that is established by those
"ignored" messages cannot be ignored without the state in the two peers
becoming unsynchronized.]
"""

p2: section 5.3.2 (Accept header)

Osama Mazahir — 2013-05-09T00:36:24

In section 5.3.1, the weight is defined as:

------------

     weight = OWS ";" OWS "q=" qvalue

     qvalue = ( "0" [ "." 0*3DIGIT ] )

            / ( "1" [ "." 0*3("0") ] )

------------



In section 5.3.2, the Accept header grammar is defined as:

------------

     Accept = #( media-range [ accept-params ] )



     media-range    = ( "*/*"

                      / ( type "/" "*" )

                      / ( type "/" subtype )

                      ) *( OWS ";" OWS parameter )

     accept-params  = weight *( accept-ext )

     accept-ext     = OWS ";" OWS token [ "=" word ]

------------



The grammar seems to imply that the weight is always the first parameter.  But then the example shows that the weight is not the first parameter (e.g. the level=2 case):

------------

     Accept: text/*;q=0.3, text/html;q=0.7, text/html;level=1,

             text/html;level=2;q=0.4, */*;q=0.5

------------



Which way is correct?



Thanks,

--Osama.

Proposal: New Frame Size Text (was: Re: Design Issue: Frame Size Items)

James M Snell — 2013-05-09T00:12:38

Suggested replacement text for the current "Frame Size" discussion in
the spec...

...
While the flow control protocol and framing mechanisms defined by
this specification are largely independent of one another, the flow
control WINDOW_SIZE places an upper limit on the total amount of data
an endpoint can send to a peer at any given time. DATA, HEADERS,
HEADERS+PRIORITY and PUSH_PROMISE frame sizes MUST NOT exceed the
current WINDOW_SIZE for the stream or connection and MUST NOT be
greater than 65,535 bytes. The 8 bytes of the frame header are not
counted toward this limit.

When a new connection is established, both endpoints are permitted
to begin sending frames prior to the establishment of an initial flow
control WINDOW_SIZE. Accordingly, there is a risk that an endpoint
might initially send frames that are too large for the peer to handle.
To mitigate this risk, it is RECOMMENDED that, until the initial
WINDOW_SIZE is established, the total size of individual
header-bearing frames not exceed the current TCP Maximum Segment Size
(MSS) and that individual DATA frames are no larger than 4096 bytes.
The 8-byte frame header is included in these limits.

If an endpoint is unable to process a frame due to its size and the
frame specifies any stream identifier field value other than 0x0, the
endpoint MUST respond with a <xref target="StreamErrorHandler">stream
error</xref> using the FRAME_TOO_LARGE error code. If the stream
identifier field value is 0x0, the endpoint MUST send a <xref
target="ConnectionErrorHandler">connection error</xref> using the
FRAME_TOO_LARGE error code.
...

- James

On Wed, May 8, 2013 at 1:56 PM, Poul-Henning Kamp <phk-HF+kCXwiSi60G1uiEUka0Q< at >public.gmane.org> wrote:

redundancy with WINDOW_UPDATE and SETTINGS_FLOW_CONTROL_OPTIONS

William Chan (陈智昌 — 2013-05-08T18:56:14

There's a WINDOW_UPDATE flag (
http://http2.github.io/http2-spec/#EndFlowControl) for ending flow control
for the session or individual stream. There's also
a SETTINGS_FLOW_CONTROL_OPTIONS for ending flow control for the session and
for all streams.

I think all these knobs are useful, but the redundancy is unnecessary. How
about we make SETTINGS_FLOW_CONTROL_OPTIONS specific to stream flow control
and remove the session flag? To disable flow control for the session, use
WINDOW_UPDATE with the end flow control flag and stream id 0.

http://http2.github.io/http2-spec/#rfc.section.3.1 connections

William Chan (陈智昌 — 2013-05-08T18:46:06

"""
The HTTP/2.0 connection is an Application Level protocol running on top of
a TCP connection ([RFC0793] <http://http2.github.io/http2-spec/#RFC0793>).
The client is the TCP connection initiator.
"""

I'm concerned this text confuses people, since HTTP/2 can run in many
different fashions:
- directly over TCP
- over TLS
- tunneled (for example, we have production Chromium code that supports
tunneling SPDY over SPDY)

I think it should suffice to say that HTTP/2 runs over a reliable transport.

http://http2.github.io/http2-spec/#FrameHeader section on Flags being unset

William Chan (陈智昌 — 2013-05-08T18:26:47

"""
The remaining flags can be assigned semantics specific to the indicated
frame type. Flags that have no defined semantics for a particular frame
type MUST be ignored, and MUST be left unset (0) when sending.
"""

Is there a reason that it MUST be left unset? Why not allow extensibility
like we do for unknown frames? I actually don't think there's much value to
specifying behavior for unused bits.

Requiring proxies to process warn-date

Mark Nottingham — 2013-05-08T13:14:09

In #480, Alex brought this text up in p6:


My inclination here is to change the first paragraph to begin "If a sender generates a message...", and the second to be "If a recipient receives...", also removing "forwarding" later down.

This is because IME proxies do not do any of this for messages that they aren't caching, and moreover there are whole classes of implementations that won't.

Thoughts?

--
Mark Nottingham   http://www.mnot.net/