SMTP and STARTTLS: how mail between servers is encrypted

When your mail server delivers a message to another domain, it looks up the recipient’s MX records and opens an SMTP connection to one of those hosts on port 25. SMTP itself is plain text. Encryption is added on top with STARTTLS (RFC 3207): the two servers start in cleartext and then agree to switch the same connection to TLS.

That works well against someone passively listening on the network. Against an attacker who can modify traffic it is much weaker — unless the receiving domain publishes MTA-STS or DANE.

How a STARTTLS session looks

S: 220 mx1.example.net ESMTP
C: EHLO sender.example.org
S: 250-mx1.example.net
S: 250-PIPELINING
S: 250-SIZE 52428800
S: 250-STARTTLS
S: 250 8BITMIME
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
   (TLS handshake: version, cipher, server certificate)
C: EHLO sender.example.org
S: 250-mx1.example.net
S: 250-PIPELINING
S: 250-SIZE 52428800
S: 250 8BITMIME
C: MAIL FROM:<alice@example.org>

1. The server greets the client with a 220 banner.
2. The client sends EHLO; the server lists its extensions. STARTTLS among them means TLS is available.
3. The client sends STARTTLS, the server answers 220, and the TLS handshake runs on the same TCP connection.
4. The session starts over inside TLS. Both sides must forget everything learned before the handshake, the client sends EHLO again, and the server no longer offers STARTTLS (RFC 3207, section 4.2).

This guide is about server-to-server delivery on port 25. When an application or mail client submits mail to its provider, it uses port 587 with STARTTLS or port 465 with TLS from the first byte, together with authentication; RFC 8314 recommends the latter for submission.

Why it is “opportunistic”

Nothing in an email address or an MX record says whether the receiving server supports TLS. So senders use TLS when the server offers it and deliver in cleartext when it doesn’t. RFC 3207 even forbids a public MX server from requiring STARTTLS, so that servers without TLS can still deliver mail to it.

This model is called opportunistic security (RFC 7435): encrypt whenever possible, authenticate when possible. RFC 7435 also notes that it is common for a sending server to retry in cleartext when the TLS handshake fails.

What an active attacker can do

STARTTLS defeats passive eavesdropping. An attacker who can change traffic between the servers has several options, described in RFC 8461 and RFC 7672:

• Strip STARTTLS. Everything before the handshake is unprotected. Deleting the 250-STARTTLS line from the EHLO response makes the sender believe TLS isn’t available, and it sends the message in cleartext.
• Break the handshake. Tampering with the STARTTLS command or the handshake makes it fail, and many senders fall back to cleartext.
• Redirect the connection. Without DNSSEC, forged MX or address answers send the sender to the attacker’s server, which happily offers TLS with its own certificate.
• Present a different certificate. If the sender doesn’t validate the certificate, an interceptor in the middle can terminate TLS and read everything.

Usually nobody notices: the mail arrives as usual.

Certificate validation in practice

Browsers reject a certificate that doesn’t match the site name. Opportunistic SMTP generally doesn’t, for reasons explained in RFC 7672, section 1.3:

• the sender has no trustworthy name to check against — the MX host name comes from DNS, which an attacker can forge without DNSSEC;
• many receiving servers use self-signed certificates or certificates from private CAs, so sending servers are rarely set up to validate;
• with nothing telling it which certificate to expect, a sender that rejected such certificates would only bounce mail or fall back to cleartext — worse than encrypting without authentication (RFC 7435).

So with plain STARTTLS, a self-signed or mismatched certificate usually still gets you encryption — but no protection against an active attacker. Keep a valid certificate anyway: MTA-STS requires one, and RFC 7435 observes that some senders accept self-signed certificates but not expired ones, and then fall back to cleartext.

The fixes: MTA-STS and DANE

Both let a receiving domain declare “TLS is available here and this is how to authenticate it”, so senders that support them refuse to deliver over a downgraded or forged connection.

	MTA-STS (RFC 8461)	DANE (RFC 7672)
Published as	TXT record _mta-sts.<domain> plus a policy file on https://mta-sts.<domain>/.well-known/mta-sts.txt	TLSA records at _25._tcp.<MX host>
Trust based on	Public certificate authorities	DNSSEC
Certificate must	Chain to a trusted CA, be unexpired and contain the MX host name	Match the TLSA record
Main limitation	First contact: a sender without a cached policy can be kept from fetching it	Requires DNSSEC-signed zones and senders that validate DNSSEC
Soft start	mode: testing	Publish TLSA records only for some MX hosts

MTA-STS is announced by a TXT record that carries the policy version:

_mta-sts.example.com.  3600  IN  TXT  "v=STSv1; id=20261024T120000"

The policy file at https://mta-sts.example.com/.well-known/mta-sts.txt lists the MX host names that may receive mail and how long senders should remember the policy:

version: STSv1
mode: enforce
mx: mx1.example.net
mx: mx2.example.net
max_age: 604800

A DANE record pins the server’s key. RFC 7672 recommends the 3 1 1 form — a SHA-256 hash of the server’s public key — and with it the certificate’s names and expiry date are ignored; the key published in DNS is what counts (section 3.1.1):

_25._tcp.mx1.example.net.  3600  IN  TLSA  3 1 1 <SHA-256 hash of the public key>

The two can coexist; a sender that supports both must not let MTA-STS override a failed DANE check. Whichever you deploy, add a TLS-RPT record (RFC 8460) so that senders report failed TLS connections to you. Step-by-step setup is in MTA-STS and TLS-RPT.

TLS versions

RFC 8996 deprecates TLS 1.0 and 1.1: they must not be used. The current best practice, RFC 9325, requires support for TLS 1.2 and recommends TLS 1.3, preferred when both sides support it. MTA-STS also requires participating servers to support TLS 1.2 or later (RFC 8461, section 7.2).

For a public MX server, offer TLS 1.2 and 1.3. If it negotiates only 1.0 or 1.1, the TLS library or the mail server software is out of date. RFC 9325, section 5.2 adds a caveat for opportunistic protocols: strict settings can cause fallback to cleartext, which is worse than an outdated TLS version. A very old sender that can’t do TLS 1.2 will usually deliver to your MX unencrypted rather than not at all — check your mail server logs for such connections before removing old versions.

What the SMTP and TLS checker shows

The SMTP and TLS checker connects to your domain’s MX hosts on port 25, as a sending server would, and reports:

• the banner — whether the server answers and with which greeting;
• the EHLO response and the extensions offered;
• whether STARTTLS is offered and the upgrade succeeds;
• the negotiated TLS version;
• the certificate — whether it covers the MX host name, and when it expires.

It ends the session after the TLS handshake and does not send mail: no MAIL FROM or RCPT TO, so it says nothing about whether a mailbox exists.

Finding	Effect	What to do
No STARTTLS in the EHLO response	All incoming mail arrives in cleartext; MTA-STS can’t be enforced	Enable TLS in the mail server and install a certificate
STARTTLS offered, handshake fails	Senders retry in cleartext or defer mail	Check the certificate, key and TLS settings
Only TLS 1.0 or 1.1 negotiated	Outdated software; up-to-date senders refuse these versions, so TLS fails	Update the TLS library and mail server; enable TLS 1.2 and 1.3
Certificate doesn’t cover the MX host name	Fails MTA-STS validation	Get a certificate whose SAN includes the MX host name
Certificate expired or about to expire	Fails MTA-STS validation; some senders fall back to cleartext	Renew it, preferably automatically

If your mail is hosted by a provider, the MX hosts and their certificates are the provider’s responsibility, but you can still publish MTA-STS and TLS-RPT for your domain. Check the policy with the MTA-STS checker and the reporting record with the TLS-RPT checker.

Why port 25 may be blocked for you

Testing from a laptop often fails before SMTP even starts. Many residential and mobile ISPs block outgoing connections to port 25 to stop infected computers from sending spam, and so do cloud platforms: Amazon EC2 blocks port 25 to public addresses by default, Google Compute Engine blocks it to destinations outside the VPC network, and Azure blocks it for most subscription types. A timeout from your network doesn’t mean the MX server is down — use the checker, or test from a server that is allowed to connect out on port 25:

openssl s_client -starttls smtp -connect mx1.example.net:25 -crlf -servername mx1.example.net -verify_hostname mx1.example.net -brief

-brief prints the protocol version and whether the certificate verified for that host name. If your application needs to send mail from such a network, submit it to a relay on port 587 or 465 with authentication instead of delivering directly on port 25.