Dobrev.EU Blog

Things I want to share

DMARC or How Social Networks Are Figthting SPAM

| Comments

I received recently a notification from Yahoo that customers of the company I work for use our own SMTP servers to “forge” mails that come from their own Yahoo account.

Hello,

Yahoo Mail recently enabled a DMARC reject policy to protect users from increasing email spam that uses Yahoo users email addresses from other mail servers (you can read about it here).

We are reaching out to you regarding the domain “our domain” because we noticed a number of your customers’ emails are being rejected. This is due to the fact that these emails are from your customers with “@yahoo.com” in their “From” address and are originating from your servers. With our recent policy change, DMARC compliant systems will reject such emails.

To help you assist your customers through this change, we have outlined the remediation in your case. You should follow the applicable recommendations from those we have listed below:

Small Business Owners / ESPs / ISPs / Domain Hosting

1. If you are sending the email on behalf of a business:
a. then the customer is best served if they move to sending email from their own domain; OR
b. Use an address you control, which could be a dedicated address at your site. E.g. If you are “b2b.example” then “From: “Example Sender” <example-sender@b2b,example>”; OR
c. Use a single address for different senders E.g. If you are “b2b.example” then you could do
i. “Reply-to: “Example Sender” <example-sender@yahoo.com>”; AND
ii. “From: “Example Sender” <noreply@b2b.example>”
2. If you are an ISP or an email provider and your users want to use Yahoo addresses; then
a. Consider allowing customers to connect directly to Yahoo SMTP servers; OR
b. Contact us at dmarc-help@yahoo-inc.com to discuss authentication and configuration options.

Websites allowing visitors to share links

If your website provides the ability to share items in email, we recommend that you send these emails from your own domain. You can set a Reply-To: header with their address so that people can reply to the sharer instead of replying to you.

E.g. If you are “sharing.example”
1. Reply to: “Example Sender” <example-sender@yahoo.com>;” AND
2. From: “Example Sender” <noreply@sharing.example>

For additional details, please click here.

If you need any clarifications or further information on how to assist your customers, please feel free to reach out to dmarc-help@yahoo-inc.com

Thank you
The Yahoo Mail Team

Long story short – our customers use their Yahoo accounts in From: field when sending mails out from our shop-applications to their customers over our SMTP servers and this makes Yahoo unhappy about it, because they’re the only company allowed to do that from their own SMTP servers. But how did they came to this conclusion and decission to mail us about the problem? One term was unknown to me up to this mail: DMARC

What is DMARC?

DMARC.org was established not long time ago. DMARC stands for “Domain-based Message Authentication, Reporting & Conformance” and it allows domain owners to publish policy statements in DNS telling receiver domains what to do with messages that don’t authenticate. On its official website, DMARC is described as standardizing “how email receivers perform email authentication using the well-known SPF and DKIM mechanisms.”

This is cool! Why don’t I have it?

I’m using SPF and DKIM for my SMTP server since the day I bought this domain which I managed to pre-order before .EU-TLD was launched. So adding DMARC on top of them should not be a problem. Or at least I thought so…

My current setup

  • Ubuntu 12.04
  • Postfix
  • Amavisd-new (for anti-spam and anti-virus checks) and DKIM signing
  • ClamAV
  • Dovecot /for IMAP(S) and POP3(S)/
  • LDAP /for virtual domains and users management/
  • Bind9 DNS /master-slave/

Existing mail flow diagramm

The simplest I can put the flow of an incoming message is:

Internet => SMTPD => Amavisd-new => SMTPD => Delivery => Virtual Table => Local storage

Required changes

In order to enable DMARC I need to grab the results from a valid SPF and/or DKIM check that my server did. There are at least 2 ways of achieving that goal one mainly adds new services in /etc/postfix/master.cf the second implies the usage of Sendmail milters. Why milters? Because milters are effective way to do Header checks of incoming messages even before the DATA phase and most importantly they are able to do changes to the content of the headers while they do their job. So milters are my choice then. I can set their order in Postfix and ensure that my DMARC test won’t be executed before SPF or DKIM. So I ended up with:

Internet => SMTPD => SPF Check => DKIM check => DMARC check => Amavisd-new => SMTPD => Delivery => Virtual Table => Local storage

Required software

So far I was happy with the way Amavis was dealing with DKIM signing and checks. But I can’t continue using it for that purpose because policy services in Postfix are executed after the milters (I came to that conclusion after few hours of debugging and countless mails that I had to send). For that reason I removed DKIM settings from Amavis and started looking for milters that can help to achieve my goal. I’d like to spare you the time I spent to finally come up with the following set of software:

smf-spf

Download the sources from Source-Forge. As I already wrote this project is marked as inactive so I’m linking a local copy aswell.

[smf-spf Installation]
1
2
3
4
5
tar zxvf smf-spf-2.0.2.tar.gz
cd smf-spf-2.0.2
./configure
make
make install

If you receive an error that libspf2 is missing although you installed it then edit Makefile and add -L/usr/lib/libspf2 to LDFLAGS

The configuration is really easy. All I had to do is set the socket for my SPF checker to run at.

/etc/mail/smf-spf/smf-spf.conf (smf-spf.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
# /etc/mail/smfs/smf-spf.conf
#
# smf-spf configuration file v2.0.2 (it's read at start)
#

# Whitelist by a sender IP address
#
# The syntax is an IP address followed by a slash
# and a CIDR netmask (if the netmask is omitted, /32 is assumed)
#
WhitelistIP   127.0.0.0/8
WhitelistIP   10.0.0.0/8
WhitelistIP   172.16.0.0/12
WhitelistIP   192.168.0.0/16

# Whitelist by a sender PTR record (reverse DNS record)
#
# Performs a case insensitive substring match
#
#WhitelistPTR .friendlydomain.tld
#WhitelistPTR friendlyhost.friendlydomain.tld

# Whitelist by an envelope sender e-Mail address
#
# Performs a case insensitive substring match
#
#WhitelistFrom    friend@
#WhitelistFrom    @friendlydomain.tld
#WhitelistFrom    friend@friendlydomain.tld

# Whitelist by an envelope recipient e-Mail address
#
# Performs a case insensitive substring match
#
#WhitelistTo  postmaster@
#WhitelistTo  @yourspamloverdomain.tld
#WhitelistTo  spamlover@yourdomain.tld

# Refuse e-Mail messages at SPF Fail results (RFC-4408)
#
# Default: on
#
RefuseFail    on  # (on|off)

# Subject tagging of e-Mail messages at SPF SoftFail
# and Fail (if RefuseFail set to off) results
#
# Default: on
#
TagSubject    on  # (on|off)

# Subject tagging string
#
# Default: [SPF:fail]
#
Tag       [SPF:fail]

# Build a standard Received-SPF: header
#
# Default: on
#
#AddHeader    on  # (on|off)
AddHeader on
# Quarantine of e-Mail messages at SPF SoftFail
# and Fail (if RefuseFail set to off) results
#
# Default: off
#
#Quarantine   off # (on|off)

# Quarantine mailbox
#
# Default: postmaster
#
#QuarantineBox    postmaster
#QuarantineBox    spambox@yourdomain.tld

# In-memory cache engine TTL settings
#
# The time is given in seconds, except if a unit is given:
# m for minutes, h for hours, and d for days
# Specify zero to disable caching
#
# Default: 1h
#
TTL       1h

# Run as a selected user (smf-spf must be started by root)
#
# Default: smfs
#
User      smfs

# Socket used to communicate with Sendmail daemon
#
# Default: unix:/var/run/smfs/smf-spf.sock
#
#Socket       unix:/var/run/smfs/smf-spf.sock
Socket  inet:20002@localhost

# Facility for logging via Syslog daemon
#
# Default: mail
#
Syslog        mail    # (daemon|mail|local0...local7)

OpenDKIM

OpenDKIM has 2 very nice features: it’s able to sign and verify messages. So after I removed DKIM from my Amavis implementation I can switch to it. Install OpenDKIM from the repositories:

OpenDKIM Installation
1
sudo apt-get install opendkim opendkim-tools

Then replace the contents of your configuration file with the example provided

/etc/opendkim.conf (opendkim.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# This is a basic configuration that can easily be adapted to suit a standard
# installation. For more advanced options, see opendkim.conf(5) and/or
# /usr/share/doc/opendkim/examples/opendkim.conf.sample.

# Enable Logging
Syslog yes
SyslogSuccess yes
LogWhy yes

# Required to use local socket with MTAs that access the socket as a non-
# privileged user (e.g. Postfix)
UMask         002

# Common settings. See dkim-filter.conf(5) for more information.
AutoRestart             yes
Background              yes
Canonicalization        relaxed/relaxed
DNSTimeout              5
Mode                    sv
SignatureAlgorithm      rsa-sha256
SubDomains              no
#UseASPDiscard          no
#Version                rfc4871
X-Header                no

# Sign for example.com with key in /etc/mail/dkim.key using
# selector '2007' (e.g. 2007._domainkey.example.com)
#Domain           example.com
#KeyFile      /etc/mail/dkim.key
#Selector     2007

# Commonly-used options; the commented-out versions show the defaults.
#Canonicalization simple
#SubDomains       no
#ADSPDiscard      no

# Always oversign From (sign using actual From and a null From to prevent
# malicious signatures header fields (From and/or others) between the signer
# and the verifier.  From is oversigned by default in the Debian pacakge
# because it is often the identity key used by reputation systems and thus
# somewhat security sensitive.
OversignHeaders       From

# List domains to use for RFC 6541 DKIM Authorized Third-Party Signatures
# (ATPS) (experimental)

#ATPSDomains      example.com

KeyTable /etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTable
ExternalIgnoreList /etc/opendkim/TrustedHosts
InternalHosts /etc/opendkim/TrustedHosts

# Set the user and group to opendkim user
UserID opendkim:opendkim

# Specify the working socket
Socket inet:20003@localhost

Create a key for your domain:

DKIM key creation
1
2
cd /etc/opendkim/keys/
opendkim-genkey -a -t -b 2048 -s mail -d dobrev.eu

-a – Append domain name to the DNS entry
-s – Hostname of the server
-d – Domain name
-b – Key size in bits (I’m a bit paranoid)

This command creates 2 files: hostname.txt containing the DNS RR and hostname.private containing the private domain key. Don’t forget to rename them to something meaningful in case your SMTP server hosts more than one domain.

Next you need to configure the keys for OpenDKIM to use them for signing. Open /etc/opendkim/KeyTable and enter:

/etc/opendkim/KeyTable
1
2
# KeyTable ID    File location
dobrev.eu dobrev.eu:mail:/etc/opendkim/keys/mail.dobrev.eu.private

Keytable ID is the unique ID for the key. The second column describes the “domain:hostname:path to private key”. This entry is being used to set DKIM parameters.

Then add your key to the list of signing keys:

/etc/opendkim/TrustedHosts
1
2
# Account    KeyTable ID
*@dobrev.eu   dobrev.eu

And finally /etc/opendkim/TrustedHosts:

/etc/opendkim/TrustedHosts
1
2
127.0.0.1
dobrev.eu

Restart OpenDKIM and check the logs if anything went wrong. Up to that point OpenDKIM is fully configured for signing and verification of mails. But 3rd party SMTP servers still can not verify mails that come from me. So I had to modify my DNS server and add the information from the .txt file that opendkim-genkey created for me in the previous step.

N.B. because I’m using a 2048 bit key I had to split the key in multiple lines else BIND was complaining that the RR is too long.

OpenDMARC

Install OpenDMARC:

Install OpenDMARC
1
apt-get install opendmarc

Create a configuration file:

/etc/opendmarc.conf (opendmarc.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
# This is a basic configuration that can easily be adapted to suit a standard
# installation. For more advanced options, see opendkim.conf(5) and/or
# /usr/share/doc/opendmarc/examples/opendmarc.conf.sample.

##  AuthservID (string)
##    defaults to MTA name
#
AuthservID mail.dobrev.eu

##  BaseDirectory (string)
##    default (none)
##
##  If set, instructs the filter to change to the specified directory using
##  chdir(2) before doing anything else.  This means any files referenced
##  elsewhere in the configuration file can be specified relative to this
##  directory.  It's also useful for arranging that any crash dumps will be
##  saved to a specific location.
#
#BaseDirectory /var/run/opendmarc

##  ChangeRootDirectory (string)
##    default (none)
##
##  Requests that the operating system change the effective root directory of
##  the process to the one specified here prior to beginning execution.
##  chroot(2) requires superuser access.  A warning will be generated if
##  UserID is not also set.
# 
# ChangeRootDirectory /var/chroot/opendmarc

##  ForensicReports { true | false }
##    default "false"
##
ForensicReports true

##  IgnoreHosts path
##    default (internal)
##
# IgnoreHosts /usr/local/etc/opendmarc/ignore.hosts

##  IgnoreMailFrom domain[,...]
##    default (none)
##
# IgnoreMailFrom example.com

##  PidFile path
##    default (none)
##
##  Specifies the path to a file that should be created at process start
##  containing the process ID.
##
#
PidFile /var/run/opendmarc.pid

##  RejectFailures { true | false }
##    default "false"
##
RejectFailures false

##  Socket socketspec
##    default (none)
##
##  Specifies the socket that should be established by the filter to receive
##  connections from sendmail(8) in order to provide service.  socketspec is
##  in one of two forms: local:path, which creates a UNIX domain socket at
##  the specified path, or inet:port[@host] or inet6:port[@host] which creates
##  a TCP socket on the specified port for the appropriate protocol family.
##  If the host is not given as either a hostname or an IP address, the
##  socket will be listening on all interfaces.  This option is mandatory
##  either in the configuration file or on the command line.  If an IP
##  address is used, it must be enclosed in square brackets.
#
Socket inet:20004@localhost

##  SoftwareHeader { true | false }
##    default "false"
##
##  Causes the filter to add a "DMARC-Filter" header field indicating the
##  presence of this filter in the path of the message from injection to
##  delivery.  The product's name, version, and the job ID are included in
##  the header field's contents.
#
SoftwareHeader false

##  Syslog { true | false }
##    default "false"
##
##  Log via calls to syslog(3) any interesting activity.
#
Syslog true

##  SyslogFacility facility-name
##    default "mail"
##
##  Log via calls to syslog(3) using the named facility.  The facility names
##  are the same as the ones allowed in syslog.conf(5).
#
SyslogFacility mail

##  TemporaryDirectory path
##    default /var/tmp
##
##  Specifies the directory in which temporary files should be written.
#
TemporaryDirectory /var/tmp

##  TrustedAuthservIDs string
##    default HOSTNAME
##
##  Specifies one or more "authserv-id" values to trust as relaying true
##  upstream DKIM and SPF results.  The default is to use the name of
##  the MTA processing the message.  To specify a list, separate each entry
##  with a comma.  The key word "HOSTNAME" will be replaced by the name of
##  the host running the filter as reported by the gethostname(3) function.
#
TrustedAuthservIDs HOSTNAME


##  UMask mask
##    default (none)
##
##  Requests a specific permissions mask to be used for file creation.  This
##  only really applies to creation of the socket when Socket specifies a
##  UNIX domain socket, and to the HistoryFile and PidFile (if any); temporary
##  files are normally created by the mkstemp(3) function that enforces a
##  specific file mode on creation regardless of the process umask.  See
##  umask(2) for more information.
#
UMask 0002

##  UserID user[:group]
##    default (none)
##
##  Attempts to become the specified userid before starting operations.
##  The process will be assigned all of the groups and primary group ID of
##  the named userid unless an alternate group is specified.
#
UserID opendmarc

HistoryFile /var/run/opendmarc/opendmarc.dat
ReportCommand /usr/sbin/sendmail -t -F 'Dobrev.EU DMARC Report' -f 'noreply-dmarc-reports@dobrev.eu'

Pretty much you’re done. Your server is now DMARC aware. Start the service and check the logs.

Sending out DMARC reports

Up to this moment my box is able to check incoming emails for valid SPF and/or DKIM. DMARC is then collecting statistics in /var/run/opendmarc/opendmarc.dat. I need a way to extract them in a XML format and notify all domain owners about the percentage of failed mails (or all mails that didn’t pass neither SPF nor DKIM – effectivly SPAM or forged mails). Opendmarc has a set of tools to do this. I had to patch them a bit in order to make them work. You can find what later in the Troubleshooting section.

Install the MySQL schema that came with opendmarc, a copy of it resides in /usr/share/docs/opendmarc*. Then use the script to populate the DB, prepare the reports and send them out.

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash

set +x

DBHOST="localhost"
DBUSER="opendmarc"
DBPASS="changeme"
DBNAME="opendmarc"

/usr/sbin/opendmarc-import -dbhost=${DBHOST} -dbuser=${DBUSER} -dbpasswd=${DBPASS} -dbname=${DBNAME} < /var/run/opendmarc/opendmarc.dat
cat /dev/null > /var/run/opendmarc/opendmarc.dat
/usr/sbin/opendmarc-reports -dbhost=${DBHOST} -dbuser=${DBUSER} -dbpasswd=${DBPASS} -dbname=${DBNAME} -interval=86400 -report-email "noreply-dmarc-reports@dobrev.eu" -report-org "Dobrev.EU"
/usr/sbin/opendmarc-expire -dbhost=${DBHOST} -dbuser=${DBUSER} -dbpasswd=${DBPASS} -dbname=${DBNAME}

But where do these reports go to?

Adding DMARC policy for a domain

A DMARC policy is a DNS TXT RR which tells checking servers what to do with an email in case it fails verification.

1
_dmarc.dobrev.eu. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc-rua@dobrev.eu; ruf=mailto:dmarc-ruf@dobrev.eu"

So I’m just letting receiver servers to decide alone (p=none) what to do with failed mails based on their own policies and to send statistics to either emails. Most major service providers (Facebook, Twitter, Microsoft, Google and Yahoo just to name a few) already have DMARC enabled so I’m sending out reports to them too. Once you receive a report from them you know you’re on the safe side too.

Bundle all together in Postfix

I added the following 5 lines to my main.cf

main.cf
1
2
3
4
5
# Pass SMTP messages through smf-spf, OpenDKIM and finally OpenDMARC
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:localhost:20002,inet:localhost:20003,inet:localhost:20004
non_smtpd_milters = inet:localhost:20002,inet:localhost:20003,inet:localhost:20004

In order to avoid double checks I had to append no-milters to my Return-from-Amavis definition in master.cf aswell.

master.cf
1
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings,no_milters

Restart postfix and check the logs. You should have your mail checked through the milters.

Testing proper configuration

Manual tests

Third-party tools

Comments