When setting up TLS, the certificate that is used by the SMTP service needs to meet a few requirements. If you read RFC 3207, it states the below:
The decision of whether or not to believe the authenticity of the other party in a TLS negotiation is a local matter. However, some general rules for the decisions are: - A SMTP client would probably only want to authenticate an SMTP server whose server certificate has a domain name that is the domain name that the client thought it was connecting to. - A publicly-referenced SMTP server would probably want to accept any verifiable certificate from an SMTP client, and would possibly want to put distinguishing information about the certificate in the Received header of messages that were relayed or submitted from the client.
Microsoft best practices are below:
The fully qualified Internet domain name of the server This may be different from the internal FQDN that is used between Edge Transport servers and Hub Transport servers and should match the A record that is published on the Internet (public) DNS server. This name should be entered as a CN in the SubjectName parameter of the New-ExchangeCertificate cmdlet.
All the accepted domain names of the organization Use the IncludeAcceptedDomains parameter of the New-ExchangeCertificatecmdlet to populate the Subject Alternative Name for the resulting certificate.
The FQDN for the connector if it isn't covered by either of the previous items Use the DomainName parameter of the New-ExchangeCertificate cmdlet to populate the Subject Alternative Name for the resulting certificate.
In order to ensure that we pass the checktls.com and testconnectivity.microsoft.com tests for TLS, we have used the below:
- Use a certificate from a trusted public CA e.g. GoDaddy
- Ensure that the certificate names include the MX records (if your MX records are mx1.contoso.com and mx2.contoso.com then ensure that each SMTP server has this name on the certificate that it is using)
- Set the FQDN of the SMTP server to be the same as the MX record name.