So far in this segment, we have shown you two different forms of MTA. First, we introduced you to the Postfix MTA, which is a transport agent used for routing e-mails from a mail client to or between mail servers and delivering them to the local mailboxes on the mail server using the SMTP protocol. Then we showed you another type of MTA which sometimes called an access agent and which the Dovecot program can be used for. This delivers mails from the local Postfix mailboxes to any remote mail client programs using the POP3 or IMAP protocol. Now we will introduce you to a third type of MTA, which can be termed a retrieval agent, and explain what we will use the program Fetchmail for. Nowadays, almost everybody has more than one e-mail account, from one or more different mail providers, which can be hard to maintain if you need to login to all those different webmail sites or use different accounts in your mail program. This is where Fetchmail comes into play. It is a program, running on the same server as your domain-wide Postfix mail server and which can retrieve all your different e-mails from all your different mail providers and pass them into the local user mailboxes of your Postfix MTA. Once they are stored in their appropriate place, users can access all these mails in the usual way provided by the access agent Dovecot over POP3 or IMAP. Here in this process we will show you how to install and integrate Fetchmail into your server running the Postfix MTA.
To Start With: What Do You Need?
To complete this process, you will require a working installation of the CentOS 7 operating system with root privileges, a console-based text editor of your choice, and a connection to the Internet in order to download additional packages. It is assumed that you are working through this chapter division process by process in the order that they appear and for this reason, it is expected that Postfix has been configured as a domain-wide MTA and Dovecot has been installed to provide a POP3/IMAP mail access service. In order to test Fetchmail in this process, we also need to have registered some external e-mail addresses: you need the name of the external e-mail server address and the port of your e-mail provider, as well as your user login credentials at hand. Often you can find this information from your mail provider’s Frequently Asked Questions (FAQ) section on their webpage. Also, for some e-mail addresses, you need to first enable POP3 or IMAP in your e-mail settings before you can use Fetchmail.
The Process
Fetchmail is not installed by default and for this reason we must begin by installing the necessary packages. Perform the following steps:
- To begin, log in your mail server running your Postfix server and type:
yum install fetchmail - Once installed, we will log into a system’s user account for which we want to enable Fetchmail to download external mail from an external mail provider into his local mailbox, in our example it will be the system user john: su -john. Now let’s configure Fetchmail with an external e-mail address. If your e-mail provider is called mailhost.com and it runs a POP3 server at pop.mailhost.com and IMAP on imap.mailhost.com with the username <user-name>, here (please substitute your own values) is an example command line to test connecting and fetching mails from this provider:
fetchmail pop.mailhost.com -p pop3 -u <user-name> -k -v - If you want to use IMAP with the same provider instead:
fetchmail imap.mailhost.com -p IMAP -u <user-name> -v - If the Fetchmail command was successful, all new messages will be downloaded from the server into your local mailbox in your user account.
How Does It Work?
Here in this process, we showed you how to install and test Fetchmail, which provides automated mail retrieval capabilities for any user account having a local mailbox on our Postfix server. As a result, for a client connecting to the mail server using POP3 or IMAP, the mails fetched this way look like normal incoming e-mails. Fetchmail is often used to combine and bundle all your different mail accounts into one single account, but you can also use it if your mail provider does not have a good virus or spam filter. Here you download the mails from your host’s e-mail server, then process the mails using tools such as SpamAssassin or ClamAV before sending mails to your clients.
So what did we learn from this experience?
We began this process by installing the YUM package for Fetchmail. As we wanted to set up Fetchmail for a system user’s mailbox called john, next we logged in as this user. Afterwards, we tested the Fetchmail program by running a simple command line to fetch mail from a single mail provider. As said before, for a successful login to your external mail provider, you need to know the exact login information (server address, port, username, and password, as well as the type of protocol) of the server before you can use Fetchmail.
Remember that, while some e-mail providers let the user decide if he wants to connect securely using SSL or not, some hosters such as gmail.com only allow secure connections. This means that the example command shown here in this process is likely to fail on every major e-mail provider if they don’t support POP3/IMAP access without SSL connections. Proceed to the next section in order to learn how to use Fetchmail with SSL POP3/IMAP encryption.
You should always prefer SSL encryption if your mail provider offers both. Also, some providers such as gmail.com only let the user use their services via webmail and disable POP3/IMAP service features by default; you need to enable them in your account’s settings on your provider’s website (see later).
We specified with the -p parameter which mail protocol to use with the fetchmail command. With the -u parameter, we specified the user identification to be used when logging in to the mailserver, which is completely dependent on our e-mail provider. For POP3, we applied the -k flag to ensure that the e-mails only get fetched from the server but never deleted (which is the default when using the POP3 protocol). Finally, we used -v to make the output more verbose and give us more information for our simple test. If your e-mail provider supports SSL, you also need to add a -ssl flag to the Fetchmail command as well as the root certificate of the mail server (see the next section for more information). If you run the previous command, Fetchmail will immediately start asking the mail server for any mail in the inbox on the server and download anything to your user’s local mailbox.
There's more…
In this section, we will show you how to configure Fetchmail to download all your e-mails from some real-life mail providers using POP3S, IMAPS, and the POP3 and IMAP protocols to your local mailbox on the Postfix server using a configuration file. Finally, we will show you how to automate the Fetchmail process.
Configuring Fetchmail with gmail.com and outlook.com e-mail accounts
Here we will configure the different external mail accounts which Fetchmail will download from: the popular gmail.com and outlook.com e-mail providers and a hypothetical one at my-email-server.com.
As we learned in the main process that Fetchmail processes configuration options on the command line by default, this should not be your preferred way of using Fetchmail to download your mail from different mail accounts automatically. Normally Fetchmail should be running as a service in daemon mode in the background at boot time or with a cron job and polls a list of mail servers defined in a special configuration file at specific time intervals. With this, you can conveniently configure multiple mail servers and a long list of other options.
Note
At the time of writing this book, for gmail.com to work with Fetchmail you need to login to the gmail.com website with your user account and first enable IMAP by going to your accounts settings in Forwarding and POP/IMAP. Also, enable Allow less secure apps under Sign-in & security in My account. For outlook.com, login to your mail account on the webpage, then click on options, again click on options, then click on Connect devices and apps with POP, and then click on enable POP.
Both outlook.com and gmail.com use secure POP3S and IMAPS protocols, so you need to download and install the root certificates they are signing their SSL certificates with on your Fetchmail server first in order to be able to use their services. Here we can install the Mozilla CA certification bundle, which has been compiled by the Mozilla foundation and includes the most commonly used root server certificates used by all major websites and services, such as those used by our mail providers. For gmail.com we need the Equifax Secure Certificate Authority root certificate and for outlook.com we need the root server certificate from Globalsign. Fetchmail needs these root certificates to verify the validity of any other SSL certificate downloaded from the e-mail server. Login as root on your Postfix server and install the following package:yum install ca-certificates
Afterwards, login as a Linux system user, for example, john, who we will create a new Fetchmail configuration file for, and who already has a local Postfix mailbox directory on our server located in his home directory under ~/Maildir. Now before configuring any account in the Fetchmail configuration file, you should always first test if the connection and authentication to the specific account are working with the Fetchmail command line, as shown in the previous process. For testing our different mail providers’ accounts, we need three different command line calls. For testing if your provider is using SSL encryption, you need the –ssl flag; a typical output for a mail provider who is not allowing non-SSL connections could be:Fetchmail: SSL connection failed.
Fetchmail: socket error while fetching from <userid>@<mailserver>
Fetchmail: Query status=2 (SOCKET)
If your google and outlook username is johndoe at both mail providers for testing google with the IMAPS protocol try (enter your e-mail user’s password when prompted):fetchmail imap.gmail.com -p IMAP --ssl -u johndoe@gmail.com -k -v
If the login was successful, the output should be similar to (truncated):Fetchmail: IMAP< A0002 OK johndoe@gmail.com authenticated (Success)
9 messages (2 seen) for johndoe at imap.gmail.com.
Fetchmail: IMAP> A0005 FETCH 1:9 RFC822.SIZE
For testing outlook.com with POP3S, use:fetchmail pop-mail.outlook.com -p POP3 --ssl -u johndoe@outlook.com -k -v
On success, the output should be similar to (it has been truncated):Fetchmail: POP3> USER johndoe@outlook.com
Fetchmail: POP3< +OK password required
Fetchmail: POP3< +OK mailbox has 1 messages
For our third hypothetical e-mail account at my-email-server.com, we will use POP3 or IMAP without SSL so test it using our account:fetchmail pop3.my-email-server.com -p POP3 -u johndoe -k -v
fetchmail imap.my-email-server.com -p IMAP -u johndoe -v
You should also check if all the fetched mails from your external providers have been downloaded correctly. View your system user’s local mailbox using the mailx command (mailx -f ~/Maildir). After we successfully verify that Fetchmail is able to connect to the servers and fetch some mails, we now can proceed to create a local Fetchmail configuration file in our system user’s home directory in order to automate this process and configure multiple mail addresses. Start by opening a new empty file using vi ~/.fetchmailrc. Remember that all the commands which can be put on the command line can also be used with slightly different names in the configuration file (and much more). Now put in the following content (replace john with your actual Linux system user, johndoe with your e-mail user account name, and secretpass with your actual mail password for this account):set postmaster "john"
set logfile fetchmail.log
poll imap.gmail.com with proto IMAP
user 'johndoe@gmail.com' there with password 'secretpass' is john here
ssl
fetchall
poll pop-mail.outlook.com with proto POP3
user 'johndoe@outlook.com' there with password 'secretpass' is john here
ssl
fetchall
poll pop3.my-email-server.com with proto POP3
user 'johndoe@my-email-server.com' there with password 'secretpass' is john
here
fetchall
Save and close this file. In this file, we used the following important commands:
- postmaster: Defines the local Linux user which will receive all the warning or error mails if Fetchmail runs into problems.
- logfile: Defines a filename for a log file, which can be very helpful for us to supervise and debug Fetchmail output when it’s running continuously over a long period of time in the background.
- poll section: Specifies downloading mails from a specific mail provider. For every mail account, you will define one such poll section. As you can see here, the syntax is very similar to the one used on the command line when we tested the single connections. With proto we define the mail protocol, user is the login user for the mail account, password is the login password of your account, and with the is <username> here parameter you specify which local system user account this mail account is tied to. For SSL connections you need the ssl flag, and we specified the fetchall parameter to make sure we also download all the e-mail messages flagged as read by the e-mail provider as otherwise Fetchmail would not download e-mails that have already been read.
Next change the permissions of the .fetchmailrc file because it contains passwords and should therefore not be read by anyone other than our own user:chmod 600 ~/.fetchmailrc
Finally, we execute Fetchmail with the settings given in our configuration file. For testing, we will use a very verbose parameter here: fetchmail -vvvv. All the new mails from all your different e-mail providers should now be fetched, so afterwards you should go through the output and see if every server was ready and could be polled just as the single tests we did on the command line tests earlier. All the new mails should have been downloaded to the local mailbox, so in order to read your local mails you can use the mailx command as usual, like: mail -f ~/Maildir.
Automating Fetchmail
As just said, we can now manually start the polling process every time we want by just typing in fetchmail on the command line. This will poll and fetch all new mails from the mail servers specified in our new configuration file and then after processing each entry once it will exit the program. Now what’s still missing is a mechanism to continuously query our mail servers at a specific interval updating our mailbox whenever new mails can be fetched. Here you can use two approaches. Either run the fetchmail command as a cron job or as an alternative you can start Fetchmail in daemon mode (use the parameter set daemon in your .fetchmailrc config file to activate it.) and put it in the background.
This way Fetchmail will run constantly and wake up at a given time point and start the polling until everything finishes processing and then go back to sleep until the next interval has been reached.
As both methods are basically the same, here we will show you how to run Fetchmail as a cron job, which is much easier to set up because we don’t have to create some custom systemd service files (currently in CentOS 7 there is no fetchmail systemd service available out-of-the box). For every system user (for example, john) who has a fetchmail configuration file, to start the e-mail server polling process every 10 minutes type in the following command once to register the cron job:crontab -l | { cat; echo "*/10 * * * * /usr/bin/fetchmail &> /dev/null "; } | crontab
Note
Do not set the Fetchmail polling cycle shorter than every 5 minutes; otherwise, some mail providers may block or ban you, as it just overloads their systems.
