Binary Structured HTTP Field Values

Introduction Structured Field Values for HTTP offers a set of data types for use by HTTP fields, along with a serialisation of them in a familiar textual syntax. defines an alternative, binary serialisation of those structures, and defines a mechanism for using that serialisation in HTTP/2. The primary goal of this specification is to reduce parsing overhead and associated costs, as compared to the textual representation of Structured Fields. A secondary goal is a more compact wire format in common situations. An additional goal is to enable future work on more granular field compression mechanisms.

Notational Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This specification describes wire formats using the convention described in .

Binary Structured Types This section defines a binary serialisation for Structured Field Values as defined in . A Structured Field Value can be a singular Item (such as a String, an Integer, or a Boolean, possibly with parameters), or it can be a compound type, such as a Dictionary or List, whose members are composed of those Item types. When a field value is serialised as a Binary Structured Field, each of these types is preceded by an header octet that indicates the relevant type, along with some type-specific flags. The type then determines how the value is serialised in the following octet(s). Use of each flag may not be specified by all types. When this is the case, generators MUST send 0 for them, and recipients MUST ignore them.

Literal Values A Literal Value is a special type that carries the string value of a field; they are used to carry field values that are not structured using the data types defined in . This might be because the field is not recognised as a Structured Field, or it might be because a field that is understood to be a Structured Field cannot be parsed successfully as one. A literal value's payload consists of an integer Length field (using the variable-length encoding from ), followed by that many octets of the field value. They are functionally equivalent to String Literal Representations in .

Lists A List () uses its Flags to carry a Short Member Count as three bits. If the Short Member Count is zero, it indicates that the next byte is the Member Count, represented using the variable-length encoding from . Otherwise, it indicates the Member Count itself. This allows a small list to be encoded without using an extra byte for its length. In either case, the payload indicates the number of List members that follow.

Dictionaries A Dictionary () uses its Flags to indicate its number of members, in a fashion similar to Lists. Each Dictionary member is represented by a length, followed by that many bytes of the member-key, followed by the Binary Structured Type(s) representing the member-value. A Dictionary Member MUST NOT be a Parameters (0x2).

Inner Lists An Inner List () has a payload consisting a Member Count field (using the variable-length encoding from ), followed by one or more fields representing the members of the list. The Parameters Flag indicates whether the value is followed by Parameters (see ). Parameters on the Inner List itself, if present, are serialised in a following Parameter type (); they do not form part of the payload of the Inner List (and therefore are not counted in Member Count).

Parameters Parameters () uses its Flags to indicate its number of members, in a fashion similar to Lists. Each Parameter conveys a key and a value: A parameter's fields are:

Parameter Key Length: The number of octets used for the parameter-key (using the variable-length encoding from )
Parameter Key: Parameter Key Length octets of the parameter-key
Binary Structured Type: The parameter value

The Binary Structured Type in a Parameter MUST NOT be an Inner List (0x1) or Parameters (0x2). Parameters are always associated with the Binary Structured Type that immediately preceded them. Therefore, Parameters MUST NOT be the first Binary Structured Type in a Binary Structured Field Value, and MUST NOT follow another Parameters.

Integers An Integer () has a payload consisting of a single integer (using the variable-length encoding from ). The Sign flag conveys whether the value is positive (1) or negative (0). The Parameters Flag indicates whether the value is followed by Parameters (see ).

Decimals A Decimal () has a payload consisting of two integers (using the variable-length encoding from ) that are divided to convey the decimal value. The Sign flag conveys whether the value is positive (1) or negative (0). The Parameters Flag indicates whether the value is followed by Parameters (see ).

Strings A String () has a payload consisting of an integer Length field (using the variable-length encoding from ), followed by that many octets of payload. The Parameters Flag indicates whether the value is followed by Parameters (see ). Its payload is Length octets long and ASCII-encoded.

Tokens A Token () has a payload consisting of an integer Length field (using the variable-length encoding from ), followed by that many octets of payload. The Parameters Flag indicates whether the value is followed by Parameters (see ). Its payload is Length octets long and ASCII-encoded.

Byte Sequences A Byte Sequence () has a payload consisting of an integer Length field (using the variable-length encoding from ), followed by that many octets of payload. The Parameters Flag indicates whether the value is followed by Parameters (see ). The payload is is Length octets long, containing the raw octets of the byte sequence.

Booleans A String () uses the Payload flag to indicate its value; if Payload is 0, the value is False; if Payload is 1, the value is True. The Boolean data type (type=0x8) carries its payload in the Payload Flag: The Parameters Flag indicates whether the value is followed by Parameters (see ).

Using Binary Structured Fields in HTTP/2 When both peers on a connection support this specification, they can negotiate to serialise fields that they know to be Structured Fields as binary data, rather than strings. Peers advertise and discover this support using a HTTP/2 setting defined in , and convey Binary Structured Fields in streams whose HEADERS frame uses the flag defined in .

The SETTINGS_BINARY_STRUCTURED_FIELDS Setting Advertising support for Binary Structured Fields is accomplished using a HTTP/2 setting, SETTINGS_BINARY_STRUCTURED_FIELDS (0xTODO). Receiving SETTINGS_BINARY_STRUCTURED_FIELDS with a non-zero value from a peer indicates that:

The peer supports all of the Binary Structured Types defined in .
The peer will process the BINARY_STRUCTRED HEADERS flag as defined in .
When passing the message to a downstream consumer (whether on the network or not) who does not support this extension or otherwise explicitly negotiate an equivalent mechanism, the peer will:
1. Transform all fields defined as Mapped Fields in Section 1.3 of into their unmapped forms, removing the mapped fields.
2. Serialize all fields into the appropriate form for that peer (e.g., the textual representation of Structured Fields data types defined in ).

The default value of SETTINGS_BINARY_STRUCTURED_FIELDS is 0, whereas a value of 1 indicates that this specification is supported with no further extensions. Future specifications might use values greater than one to indicate support for extensions.

The BINARY_STRUCTRED HEADERS Flag When a peer has indicated that it supports this specification as per , a sender can send the BINARY_STRUCTURED flag (0xTODO) on the HEADERS frame. This flag indicates that the HEADERS frame containing it and subsequent CONTINUATION frames on the same stream use the Binary Structured Types defined in instead of the String Literal Representation defined in to represent all field values. Field names are still serialised as String Literal Representations. In such frames, all field values MUST be sent as Binary Structured Field Values. Note that this includes Binary Literals () for those field values that are not recognised as Structured Fields, as well as textual values that cannot be successfully parsed as Structured Fields. Implementations MAY also send a field value as a Binary Literal even when it is possible to represent it as a Structured Field (e.g., for efficiency purposes). Binary Structured Field Values are stored in the HPACK dynamic table, and their lengths are used for the purposes of maintaining dynamic table size (see ). Note that HEADERS frames with and without the BINARY_STRUCTURED flag MAY be mixed on the same connection, depending on the requirements of the sender.

IANA Considerations

ISSUE: todo

Security Considerations As is so often the case, having alternative representations of data brings the potential for security weaknesses, when attackers exploit the differences between those representations and their handling. One mitigation to this risk is the strictness of parsing for both non-binary and binary Structured Fields data types, along with the "escape valve" of Binary Literals (). Therefore, implementation divergence from this strictness can have security impact.