blog.robur.coop

The Robur cooperative blog.
Back to index

SPF, DKIM, DMARC and ARC

2025-04-23

The email format is perhaps one of the oldest formats still in use for exchanging information. When this format was designed, security issues were not a primary concern, as interoperability was more important given the specific machines that existed at the time and were capable of processing emails.

However, with the advent of communication tools such as Signal, the IETF has presented a series of RFCs to enable you to "secure" your emails. This article will give a brief description of these mechanisms and their implementation in OCaml.

SPF, where comes from an email?

In the simplest case, an email that we try to send "leaves" our service and goes directly to the recipient. In this specific case, there is a mechanism that checks that an incoming connection to receive an email for a given service has an IP address associated with the sender's domain name.

For example, as a Gmail user, an email I send can only leave Gmail's servers. One of these has the IP address 173.194.0.1. This server will initiate an SMTP connection (probably with TLS) with the destination server. The destination server will then verify that, as the sender is foo@gmail.com, the current IP address attempting to send the email does indeed belong to the gmail.com service.

This verification is called SPF: Sender Policy Framework

The receiving server will simply send a series of DNS requests to gmail.com to find out whether it considers 173.194.0.1 to be an IP address belonging to the Gmail service. You can try this check manually:

$ dig +short txt gmail.com
"v=spf1 redirect=_spf.google.com"
"globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8="
$ dig +short txt _spf.google.com
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com"
"include:_netblocks3.google.com ~all"
$ dig +short txt _netblocks.google.com
"v=spf1 ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20"
"ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21"
"ip4:173.194.0.0/16 ip4:209.85.128.0/17 ip4:216.58.192.0/19"
"ip4:216.239.32.0/19 ~all"

We can see that 173.194.0.1 is indeed one of Gmail's servers (via 173.194.0.0/16). This way, you know that the email you receive is coming from a legitimate source. The information that allows you to find out the sender's IP address is, of course, stored at a much lower level than SMTP, namely the IP protocol. If you would like to learn more, check out our excellent article on this protocol, unikernels, and Miou.

More generally, RFC7208 describes the SPF language. Of all the RFCs read since RFC821, this one provides a fairly formal description of what an SPF check should be. Reading the result of an SPF query may lead to another DNS query (as was the case with redirect). It is at this point that we can ask ourselves about implementation in OCaml.

SPF, monad and OCaml

Part 4 of the RFC describes quite well all the operations that must be performed, as well as the comparisons that must be made based on the information provided by the SMTP protocol (HELO value, IP address and MAILFROM, i.e. the sender).

To enable users to perform SPF checks independently of the underlying DNS implementation (and scheduler), we can use an abstraction (as we did for ocaml-tar) consisting of a GADT that tells the user what to do (which DNS query to make) and requests the result of this operation. This is more or less a monad as described here:

type 'a record =
  | TXT : string record
  | A : Ipaddr.V4.t list record
  | AAAA : Ipaddr.V6.t list record

type 'a t =
  | Return : 'a -> 'a t
  | Reqest : Domain_name.t * 'a record * ('a -> 'b t) -> 'b t

let ( let* ) (domain_name, record) fn = Request (domain_name, record, fn)

Note here a certain "specialisation" of the monad. In general, Bind : 'a t * ('a -> 'b t) -> 'b t is more general and more commonly used in this type of case. Even if we could be satisfied with this more general case (and specialise with lwt, miou and ocaml-dns), it remains that from the user's point of view, it is impossible to introspect such a Bind: in fact, we only know that we have a value 'a that we must obtain by executing 'a t and a function requiring this value to obtain 'b t.

But we cannot know what the value 'a is. It could be an int, a string, etc. This Bind only defines a type relationship between the right side and the left side of >>=.

What we would like to be able to do is deconstruct this Bind and understand what is being asked of us: among other things, understand that at this stage, we would like to make a DNS query for a specific record to a specific domain. We will see later why it is useful to have such a restrictive Request.

The RFC then talks about stopping the process immediately as soon as a result is obtained (whether good or bad). Indeed, SPF mentions that such a framework may need to make several DNS queries — one could easily imagine an "infinite loop" of DNS queries. An unconventional way to handle this is to use exceptions to return the result.

module Result = struct
  type t =
    [ `None
    | `Neutral
    | `Pass
    | `Fail
    | `Softfail
    | `Temperror
    | `Permerror ]
end

exception Result of Result.t

let terminate result : 'a t = raise (Result result)

According to SPF's history, this framework was not really designed for a functional programming language. The mention of check_host() in the RFC suggests that this RFC is ultimately closely linked to idioms found in imperative programming (goto). check_host() has no arguments, which suggests side effects and globals. Fortunately, you can do dirty things with OCaml, and using exceptions to terminate the process and return the result works well in the end.

I mentioned the idea of an "infinite loop", particularly with the redirect action. Fortunately, the RFC describes limits (10 DNS requests) that apply throughout the process. Once again, our Request is quite specific, and implementing the idea of a loop that would attempt 10 checks is not possible. We need to extend our GADT a little to describe a list of operations we would like to attempt:

type 'a t +=
  | Tries : (unit -> unit t) list -> unit t

Note the use of unit here. In reality, at this stage, we fall back on using an exception to return the result. The overall verification process will not be Result.t t but unit t. If an exception is raised throughout the process, that is our result. Otherwise, if the process ends, the result is Neutral (as specified in paragraph [4.7 of the RFC][rfc7802-4-7]).

Finally, there is one last situation where you need to know the result of a sub-process in order to continue with other operations. For example, a check via a redirect may end with Neutral. In this case, we should try what follows (such as redirect:_spf.google.com ~all; we should then process ~all if redirect ends with Neutral).

Once again, our general Bind could satisfy us here, but we have constrained it to only make DNS requests. We therefore need to extend our GADTs once again so that, depending on the result of a process, we execute a certain new branch:

type 'a choose =
  { none : (unit -> 'a t) option
  ; neutral : (unit -> 'a t) option
  ; pass : (unit -> 'a t) option
  ; fail : (unit -> 'a t) option
  ; softfail : (unit -> 'a t) option
  ; temperror : (unit -> 'a t) option
  ; permerror : (unit -> 'a t) option }

type 'a t +=
  | Choose_on : (unit -> 'a t) * 'a choose -> 'a t

Finally, at certain points, there are some transformations from unit to unit, but in which the sequence of operations can raise the result as a Result exception. In other words, we would like Map to introduce a form of sequence of operations.

type 'a t +=
  | Map : 'a t * ('a -> 'b) -> 'b t

let ( let+ ) x fn = Map (x, fn)

At this point, we have our monad and, of course, the verification process is independent of any DNS implementation or scheduler. So there is a derivation of uspf for lwt, mirage, unix, and Miou!

DKIM and the integrity of an email

SPF is useful for verifying the origin of an email, but it does not verify the integrity of an email. When an email is transmitted, unless it is sent via TLS, it is fairly easy to alter the content of an email directly from a router that maintains TCP/IP & SMTP transmission. This is where DKIM comes into play. The idea is to be able to sign the content of the email so that the recipient can verify this signature with a public key available from the sender's email service. ocaml-dkim has been in development for quite some time. Recently, there has been talk of changing the API so that it can work with uspf, among other things.

The process for generating and verifying the signature is also the same as that offered by ARC. We therefore need to expose certain functions so that the code can be reused instead of being reimplemented. But we will come back to that later.

DKIM primarily involves generating a hash of the content of an email. This can be done by taking the content as is (simple) or in a relaxed manner. The last method requires careful reading of the RFC, as you don't want to compare hashes to find out where the bug is in your implementation and why it doesn't produce the same hash as opendkim.

Next, we need to calculate the hash of several fields (From, To, Subject, etc.) as well as our incomplete DKIM-Signature field. The latter will contain several pieces of information, including the hash of the email content, but it will not contain the signature. We will therefore calculate a hash of a partial DKIM-Signature.

When parsers matter

DKIM is perhaps the ultimate test for our Mr. MIME parser. Any alteration of the content by Mr. MIME can have dramatic consequences, as it would change the hash and signature produced by ocaml-dkim. That is why considerable work has been done on the isomorphism between the encoder and the Mr. MIME parser.

But what remains most essential are the values of our fields and how they are represented as OCaml values. In the maze of email-related RFCs, we can extract a very general "form" for all our field values. There is, of course, the email address (for From) and the date (for Date), but all of these can normally be represented in the form: unstructured.

This is how we developed unstrctrd, which is a clever mix of ocamllex, uutf and angstrom in order to:

  1. have an efficient state machine to parse this form
  2. handle UTF-8 characters
  3. compose this parser with others via angstrom

ocaml-dkim will strictly manipulate only this field format to generate the signature because unstrctrd has no loss (it does not ignore spaces, for example) and is just as capable of handling Comment-Folding-WhiteSpace (the source of all my woes).

Among other things, this allows us to avoid using certain "tricks" (such as regexes) to verify or sign an email. We stay fairly close to the format of an email, which allows us to correctly verify and generate signatures.

Results

As mentioned above, our work also involves extending the blaze tool to enable email manipulation. Here is an example of how to sign and verify an email:

$ blaze rand --seed foo= 16 > seed
$ blaze dkim gen --seed $(cat seed) | tail -n1 | cut -d' ' -f4 > key.pub
$ blaze dkim sign --seed $(cat seed) --selector robur2025 --hostname robur.coop
  --field subject \
  --field sender \
  mail.eml > signed.eml
$ blaze dkim verify -e 'robur2025:robur.coop:'$(cat key.pub) signed.eml
[OK]: sendgrid.info
[OK]: github.com
[OK]: robur.coop

Here, we generate a seed that will allow us to generate a private key. We then generate the public key and sign our email. In addition, we sign several fields (From, Subject, Sender) and finally verify our signature using our public key. The email has several DKIM signatures, but we can clearly see the last signature from robur.coop, which looks like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=robur.coop; 
  s=robur2025; q=dns/txt; bh=sA3a1op41wyVsPWKX29iI1yyELKRBONhmLMfINb0Qp4=; 
  h=from:subject:sender; b=KHIuX6qNXw8mLhSvp0gdgL+Vp9qUFZLkNsei10JcWZouq7Yl6fM
  AoVhip4Qj61hrGKSUJu+ZL/cgN9R5j2yfnFZK+4+bcMLLqPy4R+DW/fukKb8vsOx5b2nBD6BNZiJ
 EaE5CGw4PW0eacCdM0H66S7kVH5xkOnLIw0+6Ra4m4+rkg9682X7xwGSr0li22iFhY4JaJpHeQw9d
 kXUVI+RlvvAhYMm2mecRlun8vN0Ys47AeBbw3P4JUsxgzxOw/YfHJUFQ1IxoheBotEKANGtcjaEh0
 lEBknzuTNZfpJttBaZzSn+bBLaRqn/gtHqmJzoW1x1PufXkbMZaBpSeJEPhNgZ4b/jJfgC6TPv3OT
 RLZMz+PYUqrV7AHsp3eXP7u437JyoXHO18teWzOy1ZeUEp6Hx5T4HRAxwQCMsOJtAqAGdEQQMKhFh
 vMgslg3obaukCsv+5CFUDg3AGjlverj4se5NGwO02S5xMtPK+Vp2zDCf3nHwkzDAgDvr/lFne5d/E
 IxPVTXTLu2JkM3/ppmw6OYw5FNc4vsTM1FUx7SMvVluiAsNjGgphrOluFd4BFn5Wxu7q9KN2NjI8Y
 oX3yaWEMWkK7TNrZUo79ZYG2OMBs1MJhu2E57/j4J/aFMf/N9yoMVnVqWBTv7HS8vNKXknhpVQw3k
 J92B+qALtJPK45lHY=;

DMARC, mix them all!

DMARC was perhaps one of the most difficult RFC I have ever read. It is quite far from technical considerations and implementations such as those found in SPF and DKIM, and best describes a policy for aligning domains recognised by SPF and DKIM, thereby producing a result, the Authentication-Results field:

Authentication-Results: mx.google.com;
 dkim=pass header.i=@github.com;
 dkim=pass header.i=@sendgrid.info;
 spf=pass 
   smtp.mailfrom=bounces@sgmail.github.com;
 dmarc=pass (p=NONE dis=NONE) header.from=github.com

Then there is a whole logic of reports that should be sent according to a certain percentage of failures. ocaml-dmarc does not implement this part, and we focused more on interpreting and generating the Authentication-Results field. Thus, blaze has been extended to allow these fields to be "collected" and added according to what can be verified.

$ blaze dmarc collect mail.eml
mx.google.com: ✓ dkim (header.i=@github.com)
               ✓ dkim (header.i=@sendgrid.info)
               ✓ spf (smtp.mailfrom=bounces@sgmail.github.com)
               ✓ dmarc (header.from="github.com")
$ blaze dmarc verify --hostname omelet mail.eml -o new.eml
$ blaze dmarc collect new.eml
       omelet: ✓ spf
               ✓ dkim (header.i=@github.com header.s="s20150108"
                         header.b="SuEKjwfk")
               ✓ dkim (header.i=@sendgrid.info header.s="smtpapi"
                         header.b="iIicLeoJ")
               ✓ dmarc (header.from="github.com")
mx.google.com: ✓ dkim (header.i=@github.com)
               ✓ dkim (header.i=@sendgrid.info)
               ✓ spf (smtp.mailfrom=bounces@sgmail.github.com)
               ✓ dmarc (header.from="github.com")

Of course, as you can imagine, not everyone complies with RFC7601, so we had to change our parser in order to collect this information in the best possible way.

Composition between uspf and ocaml-dkim

What took particularly long in implementing DMARC was combining uspf and ocaml-dkim so that ocaml-dmarc would also offer an implementation independent of a DNS implementation and a scheduler.

The interface we would like for ocaml-dmarc would be:

  1. reading an email to calculate the hashes and signatures and collect the DKIM signatures to verify the integrity of the email
  2. a set of DNS queries that we would need to make in order to perform SPF verification based on information that we can obtain from any TCP/IP & SMTP layer

uspf is abstracted from the underlying transmission protocol as well as from the scheduler used. The same is true for ocaml-dkim. Therefore, we would like a dbuenzli-style interface:

type 'a record =
  | TXT : string record
  | A : Ipaddr.V4.t list record
  | AAAA : Ipaddr.V6.t list record

module Verify : sig
  type decoder
  type key = Key : 'a record -> key

  val decoder : ?ctx:Uspf.ctx -> unit -> decoder

  val decode : decoder ->
    [ `Await of decoder
    | `Query of decoder * Domain_name.t * key
    | `Results of (results * Domain_name.t) list
    | error ]

  val src : decoder -> string -> int -> int -> decoder
  val response : decoder -> 'a record -> 'a -> decoder
end

However, achieving such an interface requires something essential, particularly with regard to uspf: the ability to introspect the DNS query requested in order to switch from 'a Uspf.t to Verify.decode and, in particular, to Query (decoder, domain_name, record). This is why the overly general (but potentially useful) Bind introduced at the beginning of this article was not possible, because we needed more than just a relationship between a value and a function; we needed a concrete (rather than abstract) value and its function.

With DKIM, the DNS query always remains the same: dig txt selector._domainkey.gmail.com.

uspf was therefore designed for ocaml-dmarc so that ocaml-dmarc could also be independent of a DNS implementation and a scheduler. This work is similar to what ocaml-tls introduced with pass-by-value. This inevitably requires glue afterwards (for lwt or Miou) but remains more interesting in the long run.

Are emails secure?

At this point, we might legitimately think that by verifying the origin of an email via SPF, the integrity of an email via DKIM, and the alignment of the domains used for SPF and DKIM verification via DMARC, we should have a complete "stack" for email verification. However, this overlooks the common cases that arise in email transmission.

As mentioned for SPF, the "simple" case is still someone wanting to send an email directly to another person. In this context, and as we said in our last article, SPF and DKIM are sufficient and we are able to successfully transmit emails between ourselves in OCaml.

However, when it comes to mailing lists, there is an additional relay that receives emails and forwards them to other people.

Let's imagine that foo@gmail.com wants to send an email to the mailing list unikernel@robur.coop. This mailing list would then want to forward the email to bar@outlook.com.

In this forwarding, the SPF check by outlook.com would not work because the email would be sent by foo@gmail.com but the IP address sending the email would be that of robur.coop. There is a way to lie to an SMTP server by specifying a different MAILFROM. In other words, robur.coop would tell the Outlook server that the email came from unikernel@robur.coop instead of foo@gmail.com. SPF verification could work in this case because it would verify that the IP communicating with outlook.com is indeed that of robur.coop.

However, this overlooks DMARC and domain alignment. DMARC requires that the domain used for SPF verification be the same as the one found in the email address in the From field (which would therefore always be foo@gmail.com) and the one used for DKIM verification. Gmail does add a DKIM signature, but there would be a misalignment between robur.coop and gmail.com.

There is another solution often used for mailing list servers, which consists of... modifying the email. Specifically, modifying the From field and adding a DKIM signature to ensure alignment with at least one DKIM signature. This is what mailing lists still do today. However, modifying incoming emails is out of the question for us.

This is where ARC comes in, our final piece of the puzzle for email verification.

Email blockchain

ARC is a recent RFC that consists of a mailing list adding its checks (SPF, DKIM and DMARC) in the form of an Arc-Authentication-Results, as well as a new "DKIM-style" signature with Arc-Message-Signature, and finally a signature of these two fields as well as any previous ARCs added by other services.

The idea is that a relay adds a new block (called a new ARC-set) to a verification chain so that the transmission of an email via several relays can be verified at the end of the chain. In our example for bar@outlook.com, as we have seen, an SPF verification would not be possible, and if we really wanted to be SPF and DMARC compliant, this would inevitably lead to a modification of the email (and DKIM will fail).

Thanks to ARC, we can verify that the relay itself performed an SPF, DKIM and DMARC check and that it was valid. outlook.com will therefore check the new set added by the relay robur.coop (including the result of the SPF, DKIM and DMARC checks) and, provided that the relay is trusted, consider that the email transmission chain is valid and that the email does indeed come from foo@gmail.com, even though it passed through robur.coop.

ARC is actually the ultimate combination of SPF, DKIM and DMARC. And it is required by outlook.com and gmail.com. So we developed ocaml-arc and, as always, extended our blaze tool to be able to verify and sign emails:

$ blaze hdr -h From mail.eml
From: Ferry Toth <fntoth@gmail.com>
$ blaze arc verify mail.eml
fntoth@gmail.com -✓-> 01:webhostingserver.nl -✓-> 02:webhostingserver.nl
  -✓-> 03:subspace.kernel.org
$ blaze rand --seed foo= 16 > seed
$ blaze dkim gen --seed $(cat seed) | tail -n1 | cut -d' ' -f4 > key.pub
$ blaze arc sign --seed $(cat seed) \
  --seal-selector robur2025 \
  --signature-selector robur2025 \
  --hostname robur.coop \
  mail.eml > new.eml
$ blaze arc verify -e 'robur2025:robur.coop:'$(cat key.pub) new.eml
fntoth@gmail.com -✓-> 01:webhostingserver.nl -✓-> 02:webhostingserver.nl
  -✓-> 03:subspace.kernel.org -✓-> 04:robur.coop

Here, the email comes from lkml and has had several sets (from webhostingserver.nl and subspace.kernel.org). As with DKIM, we generate a seed that will be our private key. Then we extract the public key and finally sign the email (as blaze dkim). In addition to blaze dkim, blaze arc will perform the various SPF, DKIM and DMARC checks and create a new "block". We can then verify the email and it tells us that the chain is correct.

What is particularly interesting is that the SPF and DMARC checks did not work:

$ blaze hdr -h ARC-Authentication-Results new.eml
ARC-Authentication-Results: i=4; robur.coop; spf=fail smtp.helo=@robur.coop;
  dmarc=fail (p=NONE sp=QUARANTINE) header.from=gmail.com

It should also be noted that the email does not have a DKIM signature, but in this case, ARC-Message-Signature verifies integrity for us. However, ARC's interest is not in validating this information, but rather in validating the chain. Furthermore, only the first set is of interest, and it shows certain checks such as authentication and IP reversal.

$ blaze hdr -h $(seq -s ':' 4 | sed -E 's/[^:]+/ARC-Authentication-Results/g') \
  new.eml | tail -n1
ARC-Authentication-Results: i=1; webhostingserver.nl;
  smtp.remote-ip=178.250.146.69;
  iprev=pass smtp.remote-ip=178.250.146.69;
  auth=pass (PLAIN) smtp.auth=ferry.toth@elsinga.info;
  spf=softfail smtp.mailfrom=gmail.com;
  dmarc=skipped header.from=gmail.com;
  arc=none

Another piece of information is the hash calculation of the email content. It can be seen that the hash changes between the different ARC sets. This shows that the email has been modified. The same may be true for the hash of the fields. Once again, the point is not to invalidate the hash given by these sets and what we manage to produce. The point is that the last set (the one from robur.coop) does indeed have a hash that corresponds to the content of the actual email and the fields.

Furthermore, it is possible to find out who changed the email!

Conclusion

At this point, we can say that OCaml now has a complete stack for email verification. ARC was the last piece missing from our work to create a mailing list. We will finally be able to send emails and be accepted by services such as gmail.com.

What should be noted is all the work required to produce libraries that can be used by any scheduler (such as Miou) and any DNS implementation (such as ocaml-dns or your system's resolver). As we said, the saturation of the number of schedulers for OCaml 5 (to which we contribute via Miou) is not a issue for us.

This work also involves a lot of reading RFCs. This article compiles quite a few of them, even omitting those concerning emails... The reading experience we have gained in implementing protocols and formats now allows us to assimilate all these standards fairly easily.

The next step regarding emails and PTT will surely involve (re)setting up our mailing list as well as... an email search engine from our archive system!