Agreements To Fix Forwarding

via L. Anelli 13 20122 Milano MI Italy vesely@tana.it

ARC DMARC DKIM Mailing List MLM Forwarding The widespread adoption of Domain-based Message Authentication, Reporting, and Conformance (DMARC) causes problems to indirect mail flows. This document proposes a protocol to establish forwarding agreements whereby a mailbox provider can whitelist indirect mail flows directed toward its users by maintaining per-user lists of agreements.

Introduction The ability to send messages to anyone without prior agreement is a feature at the base of the success of Internet mail. It also exposes a way to abuse, though. Domain-based email authentication can limit abusive behavior by enabling receivers to unambiguously attribute responsibility of messages, and thereby to develop domain reputation. However, indirect mail flows, such as mailing lists and aliases are seriously hindered by such authentication practices (see ). Although they constitute a minor part of the global mail traffic, such hindrance poses a problem, which is what the present protocol attempts to fix. Many mailing lists break DomainKeys Identified Mail (DKIM) signatures () by adding a subject tag or a message footer. External aliases break Sender Policy Framework (SPF) () if forwarders don't change the bounce address, or break its alignment with the From: header field if they do. In both cases, they break Domain-based Message Authentication, Reporting, and Conformance (DMARC) () if there is no DKIM signature. Rewriting the very From: field, an operation known as From: munging, is a sender side solution to this problem, adopted by most mailing lists; see for a detailed description. Aliases, instead, can be avoided by having the target pull messages using a mail retrieval protocol rather than pushing them via SMTP. As an alternative, this document proposes a cooperative method that hinges on the peculiarity that both of these forwarding methods require a prior setup at the sender. At that stage, it does not cause impediment to also set up an agreement with the receiver.

Presentation by example Let's consider the procedure whereby alice@example.com subscribes to the mailing list participants@lists.example.org. The first three steps are a well known practice known as confirmed opt-in (COI). The extra steps are introduced by this protocol:

Alice subscribes to the list, either by visiting example.org web site or by sending a request to the list manager.
The mailing list manager (MLM) sends a message to Alice, asking whether she agrees to subscribe.
Alice confirms the subscription.
The MLM looks up _fixforwarding.example.com on the DNS to check if Alice's provider supports forwarding agreements (see for the TXT record format). If found, the MLM applies for a permission to send mailing list messages that may fail DMARC checks (see for the web form fields).
Upon receiving the MLM request, example.com sends a message to its user Alice, asking whether she agrees to receive that specific mail flow.
Alice confirms to her provider too.
Example.com adds participants@lists.example.org to the list of forwarders that Alice agreed upon. Then, it confirms to the MLM that the agreement is set up.
The MLM modifies Alice's subscription options removing From: munging for messages sent to her.
Example.com recognizes messages that belong to this mail flow by (1) authenticating example.org and (2) checking the List-ID: header field. Since this matches Alice's exemption list, DMARC failures can be safely ignored.

Please note that these are very simple steps, requiring very simple software. Steps and are going to be removed when the mechanism works as it should. The protocol is designed in such a way that it can be adopted by large and tiny providers alike. Also note that step requires the MLM to be able to configure From: munging on a per-subscriber basis; proposes a method to achieve such effect. Step requires the DMARC verifier to be able to perform per-user exemptions as described in Steps to are discussed in detail in the following sections.

Terms Definitions

forwarder: The entity that routinely forwards messages it has received to various recipients. Forwarders are roughly divided into two categories, aliases and mailing lists, much like SMTP definition ().
alias: A way of forwarding to a fixed set of recipients. The bounce address may or may not be changed.
mailing list: A mechanism whereby subscribers can dynamically add or remove themselves from the list. The bounce address is always changed.
bounce address: Also called envelope return address, is the parameter of the SMTP MAIL command.
receiving domain: Is the domain, or, better, its mail server(s), which receive the messages sent by the relevant forwarder.

Signers and verifiers are defined by DKIM (). The use of the term Mailing List Manager, almost always abbreviated MLM follows . A MLM is a kind of Mediator in parlance. It is usually composed of a Message Transfer Agent (MTA) with the usual suite of tools plus the mailing list specific software. Message is defined in . It consists of a header made up of one or more fields, and a body possibly composed of various MIME entities, the latter being defined in and companions. We use colon (:) to indicate header field names, as in From:, To: and the like. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Note About Rewriting the Bounce Address This section is about alias forwarding, pinpointing in which cases it is necessary to change the bounce address. Traditionally, alias expansion re-injects messages in the mail system changing just the envelope recipient address. That is, without rewriting the bounce address. This is what SMTP prescribes. However, from an authentication perspective, this doesn't always work, because some MTAs reject SPF failures —those matching a -all directive (see SPF spec, )— during the early stages of the SMTP protocol; that is, before DKIM or ARC can be verified. It is handy to leave the bounce address intact in order to have the message author notified in case of failed delivery, which increases email reliability. However, some types of errors, such as a failed delivery because the destination mailbox has been deleted, should be sent to someone who deletes the forwarding configuration. A broken alias which unflaggingly generates bounces to any bounce address is akin to an open relay. Albeit limited to non-delivery notifications, it can be exploited for spamming. To avoid this role, a receiver can carefully verify the bounce address before accepting a message, so expect your target to behave accordingly. We recommend that the bounce be replaced with the internal address of a human or robot that can figure out what to do; that can be done using a sender rewriting scheme (). However, this protocol does provide for a way to enter into agreements that cover traditional alias expansion. According to , domains consult a whitelist if they reject SPF "fail" messages before receiving the message data. If the whitelist they consult is a public DNS-based whitelist (DNSWL, ) and if it includes the forwarder's IP address, then traditional alias expansion, which does not change the bounce address can work even in the face of strict SPF policies. To recap, an MTA may check SPF after MAIL and before DATA. If the result is "fail", the MTA consults one or more DNSWLs. If the sending IP address is whitelisted, the MTA delays the rejection decision until a later stage, when the message content is received and verified via DKIM or ARC. Therefore, an agreement involving traditional alias expansion is possible if one of the following two conditions is true:

The target MXes consult public DNSWLs and the forwarder is subscribed to at least one of them, or
the target MXes never reject before receiving and evaluating the message content.

Whether and which of these conditions are met can be determined for each agreement based on the dnswl= tag declaration, as described in . In cases where none of these conditions are valid, the forwarder MUST change the bounce address.

Underscored FixForwarding DNS Resource Record Receiving domains participating in this protocol notify it to all interested parties by publishing an "underscored" resource record named _fixforwarding. According to , this record is labeled as a subdomain of the domain it refers, which is the domain-part of the pertinent email addresses. The record contains the conditions that an applicant MUST meet before applying for an agreement as well as a URI to send requests to. Requests details are specified in . The format of the _fixforwarding record follows the extensible "tag-value" syntax for DNS-based key records defined in DKIM . The folding white space terminal, FWS, is also defined there. The following tags are defined here:

auth: Authentication method used to identify the forwarder's domain; OPTIONAL, by default the method is ARC (, but DKIM is allowed as an alternative. A forwarder MUST be able to produce the required signatures before applying. ABNF: rec-auth-tag = %s"auth" [FWS] "=" [FWS] rec-auth-tag-method rec-auth-tag-method = %s"arc" / %s"dkim"
dnswl: Either a comma-separated list of DNSWLs (see ) that the domain's MXes consult before rejecting SPF failures at the early stages of SMTP transactions, or one of the keywords "none" and "all"; OPTIONAL, by default "none". See . "none" means this option is not available; that is, forwarders MUST change the bounce address to an SPF valid one. The "all" keyword means the opposite; that is, forwarders MAY leave the original bounce address intact because the receiving domain's MXes never reject messages before verifying their signatures. One or more domains mean the conditional acceptance; that is, if the DNS name that results by prepending the reverse IP address of the forwarder to one of those domains resolves to an A IP address when queried, then the message is whitelisted. ABNF: rec-dnswl-tag = %s"dnswl" [FWS] "=" [FWS] rec-dnswl-tag-opt rec-dnswl-tag-opt = rec-dnswl-tag-list / rec-dnswl-tag-none / rec-dnswl-tag-all rec-dnswl-tag-list = domain *( [FWS] "," [FWS] domain) rec-dnswl-tag-none = %s"none" rec-dnswl-tag-all = %s"all" Where domain is imported from . Each domain of the list is a DNS zone that can be queried by prepending reversed IP addresses.
post: This is the URI to post web form requests to. REQUIRED. The web format is described in . This is an HTTPS or HTTP URI.ABNF: rec-post-tag = %s"post" [FWS] "=" [FWS] rec-post-tag-uri rec-post-tag-uri = https-URI / http-URI Where https-URI and http-URI are imported from .

Forwarding Agreement Web Form Receiving domains participating in this protocol set up a web form through which forwarders can request permission to send messages that may fail DMARC checks. Forwarders who wish to enter an agreement with the receiving domain fill out the form by supplying the required values, mostly email contacts. Since there is a predefined set of fields, applicants can prepare a script which runs automatically when the form is detected by the _fixforwarding resource record, and the conditions in it are met. However, receiving domains SHOULD provide at the same post= URL an HTML form to be filled manually, so that it is possible to enter an agreement also without a dedicated script. When there are no values, the form displays empty fields for filling in; when values are submitted, it displays the result. Posted values may be encoded using either the application/x-www-form-urlencoded or the multipart/form-data media types (see ). Normal HTTP rules apply. In particular, receiving domains SHOULD check that the values posted are acceptable and return a 400 HTTP status code otherwise. If everything is fine, the HTTP server stores the values for later processing and returns status code 202. The resulting agreement status will be sent to the base address of the applicant after checks are completed. Keep in mind that this may take a human lapse of time. Organizations that need to restrict access to verified applicants MAY require an authorization, and return status code 401 if it is not present. In that case, the HTML form page SHOULD explain how generic operators can obtain the necessary authorization. This should be automatable once learned. Resorting to authorizations is justifiable only for heavily abused domains hosting myriads of users, otherwise the automated use of this protocol becomes impractical. Forwarders will likely avoid domains that are too picky about authorizations.

The Transistor Metaphor There are three email addresses that play a key role in this protocol. To give them easily memorable names, we use the metaphor of a forwarder being compared to a transistor:

collector: the entry point where the messages to be forwarded arrive,
emitter: the target address, where messages are forwarded to, which is the subject of an agreement,
base: the controlling address, where the state of the agreement is sent, starting with the result of the form submission.

Form Fields

abuse: The abuse-team or similar entity email address, where complaints about perceived forwarder's misbehavior can be sent.
agreement-id: A globally unique string that identifies a request and the related agreement. This has the syntax of a Message-ID field, defined in .
base: The forwarder's email address where the receiving domain sends the agreement acceptance, rejection, renewal or cancellation. These are simple messages readable by both machine and human, described in .
collector: The mailing list post address, or the address that the alias is attached to.
domain: The forwarder's domain to be authenticated by the receiver. It is the domain indicated in the d= tag of DKIM-Signature: or ARC-Seal: and ARC-Message-Signature: header fields, according to the auth method (). The domain MUST match the trailing part of the list-id. It is suggested that the two identifiers match as much as practical.
emitter: The user-confirmed email address that subscribed to the mailing list, or the target address of an alias. This address MUST have the same domain where the _fixforwarding record was found.
list-id: List-ID:s are naturally present in mailing list, and identify the list that the user subscribed to. For aliases, a list-id MUST be synthesized by the forwarder, for example by concatenating an encoded representation of the collector and the signing domain. This is the globally unique list-id identifier, which MUST be included in a List-Id: header field, in angle brackets as specified by , in every forwarded message. This field is needed to unequivocally identify the mail flow. The trailing part of the list-id MUST match the domain. The List-Id: header field SHOULD be signed. Failure to do so can be exploited to damage MLM reputation by replaying suitable messages.
timeout: The number of seconds that the forwarder is committed to wait for a response. The timeout expiring without a result being received is equivalent to a negative result. The timeout SHOULD be greater than 86400 seconds.
text: Free text describing the forwarding reasons or circumstances. This is intended to be presented to the user by the receiving domain when asking for confirmation. It SHOULD NOT exceed 4K octets and SHOULD NOT contain HTTP or HTTPS URIs.
token: An authorization token. HTTP provides a number of authentication and authorization schemes, see for an overview. However, not all of them are supported by all tools. This field is meant to provide a last resort possibility.

Agreement Management Agreement data consists of some settings at the receiving domain. Their correspondence to the relevant forwarding settings is handled via . Agreement data can be created in response to a web form request and removed after or when the agreement is stale. A web form request can be followed by a response or a timeout, meaning that the agreement was not agreed upon and both parties can forget about it. The forwarder MUST NOT submit the same request again, unless after a new user intervention. To honor the agreement, a DMARC filter needs two pieces of agreement data, the emitter and the list-id. For ARC, the filter checks the validity of the chain and the validity of the ARC-Message-Signature: whose d= domain matches the list-id. When oldest-pass is set, it must not be larger than the instance i= of the signer of list-id, which means that the message was received intact by the forwarder. For DKIM, a valid signature with d= matching the list-id ensures the origin of the message. Once the validity is verified, the DMARC filter checks whether the <recipient, list-id> pair matches an active agreement. If the message is then found to be part of an agreed-upon mail flow, the DMARC filter MUST exempt it from being subjected to the DMARC policy published by the message's From: domain. If the receiving domain sends aggregate reports, messages so exempted MUST be counted against the trusted_forwarder PolicyOverrideType. After setting up DMARC exemptions, the receiving domain sends an message to the forwarder's base address. From that point on, the forwarder can stop From: munging messages destined for that subscriber. The agreement lasts until it is canceled or removed by the receiving domain. It should be canceled when the corresponding mail flow stops. However, the receiving domain may ignore this, and for simplicity's sake, the forwarder has no way to cancel an existing agreement at the receiving domain, which is how agreements can become stale. The receiving domain MAY send messages to check whether the mail flow is still active. Stale agreements can then be removed. Upon receiving a new agreement request with the same coordinates as an existing agreement, the existing agreement SHOULD be considered stale and silently removed before proceeding (unless it is a concurrency bug). Conversely, a cancellation should imply removal, or at least questioning the forwarding itself, be it a mailing list or an alias.

Messages to the Base There are five types of messages that can be sent from the receiving domain to the forwarder's base address. These messages define the status of the forwarding agreement. This status is determined at the sole discretion of the receiving domain. The forwarder can respond affirmatively or negatively. In the event of a bounce or no response, the receiving domain SHOULD resend the message once or twice. After a few days without a response, the agreement can be considered stale and removed. The message types are as follows:

base-check: This message simply checks that the address exists. It can be sent after receiving a web request to confirm that it was received. It is OPTIONAL, as the HTTP return code is sufficient for this purpose. It is not an error to send it at any other time, even if it is useless
acceptance: The receiving domain accepts the agreement. From the moment this message is received onwards, messages can avoid From: munging and, if allowed by the receiving domain, avoid rewriting the bounce address.
rejection: The receiving domain, presumably after the user disavows, denies the agreement. Unauthenticated messages may then be rejected or quarantined. The forwarding itself should be questioned. A new agreement SHOULD NOT be requested again unless further interaction with the user occurs.
renewal: The receiving domain needs confirmation that this forwarding mechanism is still active. The forwarder must verify that the mechanism corresponding to the agreement is still active before responding. A bounce or no response may result in the agreement being removed.
cancellation: The agreement is canceled. For mailing lists, the user should be unsubscribed if not already done. The alias should be removed.

These messages are plain text messages with no attachments and no MIME multipart entities. The agreement-id and the message type are repeated in the Subject: and at the beginning of the body. The rest of the body MAY contain any text. ABNF: msg-subject = msg-tag WSP agreement-id ":" [WSP] msg-type msg-tag = "[FixForwarding]" msg-body = agreement-id-line msg-type-line msg-rest agreement-id-line = [CRLF] "agreement-id" ":" [WSP] agreement-id CRLF msg-type-line = [CRLF] "deal" ":" [WSP] msg-type CRLF msg-rest = *OCTET msg-type = %s"base-check" / %s"acceptance" / %s"rejection" / %s"renewal" / %s"cancellation" agreement-id = "<" id-left "@" id-right ">" subject = "Subject" ":" msg-subject CRLF message = fields CRLF msg-body Where WSP, CRLF and OCTET are imported from ; subject, id-left, id-right and fields from . Replies MUST also be text/plain, retaining the Subject: and body of the original message, possibly prefixed as is customary for Internet mail, for example by prefixing the subject with "Re:" and body text lines with "> ". Replies MUST have the In-Reply-To: and References: fields set correctly. The receiving domain can use these to match replies to the original message. For negative replies the first word in the body, and possibly the Subject: MUST be "NO". ABNF: negative-reply = fields CRLF %s"NO" CRLF msg-body Both messages and replies MUST be properly authenticated via DKIM and DMARC. Negative responses are sent when the forwarder is not committed by the agreement. This means the agreement has been previously canceled, has never been archived properly, and is in any case not active. After a negative response, the receiving domain can remove the agreement without further notice. Responses that are not negative responses are positive acknowledgements of the deal. Responses to cancellation and rejection have the same effect, regardless of whether they are positive or negative, i.e. either party may remove any reference to the agreement as a result. Negative responses to acceptance should not occur unless the web request was a forgery. Renewal messages, on the other hand, expect a negative response to eventually end the agreement.

IANA Considerations Per , please add the following entry to the "Underscored and Globally Scoped DNS Node Names" registry:

Privacy Considerations Forwarding agreements expose users' mailing list memberships as well as the final identities of forwarded messages to the administrators of their mailbox provider. This is unavoidable. However, it should be noted that such knowledge can hardly be kept secret from the people who have access to all of their mail. Users must trust their mailbox provider, and this protocol is not the only reason they should do so. However, in situations where the user wishes to hide their forwarding from the receiving domain, they SHOULD NOT request a forwarding agreement. Instead, rewriting the bounce address and From: header fields, in addition to improving deliverability, can also help hide the true origin of messages.

Security Considerations Forwarders SHOULD check DMARC and enforce author's domain's policies whenever possible. In particular, mailing lists, unless they themselves collect posts from other external mailing lists, should reject messages from domains that publish p=reject if authentication fails. Receiving domains cannot reject messages belonging to an accepted agreement, even for DMARC policy reasons reported by Arc-Authentication-Results:, because mailing lists will unsubscribe the user after a few bounces. DMARC Filtering MUST be applied by the forwarder. Failure to apply SHOULD be reported to the forwarder's abuse address.

Acknowledgments The author would like to thank Murray S. Kucherawy for reviewing the draft and Stephen J. Turnbull for devising the method reported in .

References Normative References Informative References SRS

No-Munging Method This protocol assumes that From: munging can be enabled or disabled on a per-user basis. Many MLMs instead provide an option to enable or disable it on a per-list basis. Here is a way to overcome this limitation. Let's say you have an umbrella list with two sibling sub-lists, one configured with From: munging, the other one without. Both sub-lists refuse all posts, and advertise the umbrella list as the post address. The umbrella list accepts posts according to site and list policy. It has the sibling sub-lists as its only subscribers, and won't accept more. The sibling sub-list that does From: munging accepts subscribers under the site and list policy. When forwarding agreements are accepted, the relevant subscribers are moved to the other sub-list.

Tiny Provider Example A personal or family domain that has its own mail server can participate in fixforwarding agreements by setting up a web form that mails submissions to the admin email address, which is also the base address. Then it publishes a _fixforwarding DNS record that sets the form's URL as post=. When someone submits a web request, the admin can easily check if it is good, since he or she knows each user personally. Evil submission don't really need an explicit rejection. For good ones, sending a response to the base address of the request can be done via any mail client. The same applies to renewal or cancellation, upon user request. Before sending positive responses, the admin MUST add the forwarder's data, domain and list-id, to the DMARC filter whitelist. Having a filter that accepts such a configuration is the only special software requisite. Since there are no malicious users in a home environment, this domain can decide to trust a <forwarder, list-id> pair for all users after one of them has confirmed. It is also possible to trust a forwarder domain regardless of the list-id. This type of decision simplifies the DMARC filter settings, but not agreement management.

Tiny Forwarder Example A user of the above family mail server may want some or all of her messages to be forwarded to an external address. When setting up a dot-forward file for the user, the admin can check whether the target domain has a _fixforwarding record. If so, the related web form can be filled out manually, specifying the admin's own address as the base and abuse fields. As long as renewal requests arrive every few months, they are a useful reminder; otherwise an autoresponder will be in order. The admin address can also be substituted for the bounce address of forwarded messages. This way, the user can be notified in case the remote mailbox gets full. ARC signatures are added to all forwarded messages anyway, after adding the designated List-ID: that was specified in the web form. There is not much that can be done for bounces due to DMARC policy. However, once the remote domain has accepted the forwarding request, they should not happen again.

Large Provider Example Request processing involves interaction between the receiving domain and its user who initiated the forwarding request with the forwarder. The key is to present a clear prompt to the user so that a consistent confirmation or denial is obtained. Asking the user for confirmation can also be an opportunity to manage mailbox details such as which folder or label the user wants to associate with the new mail flow. Mailbox providers can also implement a web application that lists the status of all forwarding agreements in place; this would be somewhat more reliable than monthly subscription reminders, which some call a time-distributed database. The web form should be set to reject submissions from blacklisted IP addresses or referring to known bad domains. However, it cannot use captchas. A domain that suffers an overwhelming number of bogus requests may resort to authorizations. The request for authorization may be subject to captchas or even disclosure of the requester's identity. Once granted, authorization should remain valid for all requests with the same abuse address. Accepted requests can be saved on MX servers, as <list-id> files (containing the signing domain) in each user's home folder, or as <user, list-id> pairs in a database, or in any other way convenient for the DMARC filter. Sending renewal messages to the base can be done every few months in order to clean up the storage of unused entries. Cancellation may never be used, since it is more practical to use the List-Unsubscribe: header field of a sample message to deactivate the flow. However, if the provider implements a web application that lists the status of all agreements in place, it becomes more convenient to send a cancellation message to the base to delete one of these entries.

Mailing List Example During the sunrise period, a MLM can monitor its subscribers' domains to see if any of them have adopted this protocol. A mailing list may want to check whether a _fixforwarding DNS record exists for a subscriber's domain before sending a COI. If the domain is known to handle fixforwarding requests correctly, an to the base is semantically equivalent. Otherwise, the web request can be issued after the COI has been confirmed, and the COI itself can mention that further confirmation from the provider may follow. After an acceptance mail has been received at the base, From: munging for that subscriber can be suspended. Implementing an auto-responder for the base shouldn't be difficult. The base address can be an alias for each subscriber, so that they automatically bounce after cancellation.