Irresolvable Email Deliverability Issues With Digital Ocean
The short summary to this post is as follows:
- Do not run mail servers on Digital Ocean if you are doing anything particularly important with email.
- Large and growing swathes of Digital Ocean IP addresses are blacklisted as bulk spam senders by mail providers such as Google.
- Large and growing swathes of Digital Ocean IP addresses are blacklisted by MIPSpace.
- Digital Ocean, Google, and MIPSpace will not resolve these issues for small organizations.
Digital Ocean Overview
Digital Ocean is a cheap, useful provider of cloud virtual servers, and I moved some of my sites there last year from Amazon EC2 in order to cut costs and try it out. I wasn't using much of the AWS infrastructure, so why pay for it? In Digital Ocean a virtual server is called a "droplet," and it is very easy to launch, image, destroy, and recreate droplets as needed. Every new droplet is given a random IP address from the range available to a particular hosting center, and so as is usually the case with such services an IP address can cycle rapidly through various owners.
My Email Usage
For one of the sites I moved into Digital Ocean, I run a ten year old, few thousand user advocacy newsletter. I use my own mail server for this because a few thousand users is in the range where it is much more expensive to use a service like MailChimp than it is to do a good job of sending it yourself if you already have all the code and applications set up. My newsletter is double opt-in and carefully gardened via all of the standard maintenance that has to go into being a good internet citizen when it comes to sending a bunch of mail. I know my way around this area: I even wrote a robust non-delivery report handling package back in the day. All of this is why I tend to notice even subtle deliverability issues soon after they occur, but the first one I ran into with Digital Ocean was pretty blunt.
Issue the First: Inability to Send Email to firstname.lastname@example.org
As of last year, and possibly still today, Digital Ocean use Google infrastructure to run their corporate email. Once I set up my mail server in Digital Ocean, Google bounced all my outgoing missives to email@example.com as bulk, irrespective of content. As it turned out, this was keyed to the IP address of the mail server, which was not listed in any public blacklist service at the time. The Digital Ocean response, once I got through to support on a different channel, was to recreate the mail server droplet until I obtained an IP address that worked. Which I did, and it did.
So the luck of the draw at Digital Ocean can well leave you with an IP address that is privately flagged by a service such as Google or Yahoo's anti-spam infrastructure regardless of what the blacklists say. It is essentially impossible to make that go away if you are an ordinary individual or small organization. Your only choice is to switch to a non-blocked IP address.
Issue the Second: MIPSpace
A few weeks ago, I noticed an uptick in spam-flagged non-delivery reports from a handful of newsletter recipients. On investigation I saw that my mail server IP address appeared on the MIPSpace blacklist. A little digging at Google suggested that MIPSpace is neither a helpful nor rational actor in the blacklist space, but I nonetheless went ahead with contacting both MIPSpace and Digital Ocean to see what can be done. It quickly become apparent that MIPSpace was blacklisting many Digital Ocean IP address blocks - not just my mail server IP, but thousands of IP addresses. I just happened to be in the latest large range to be added.
MIPSpace unfortunately doesn't present any useful way for small entities to be removed from their list. In my case their position is this:
Please escalate this to your upstream provider, and/or ask them to provide you with SWIP or 'rwhois' for your dedicated IP(s). We can only address reputation with the party listed in rwhois, and this service is normally provided free of charge by most hosting providers.
The relevant sections of my Digital Ocean support conversation:
At this time we are not sure if we can get this block removed. Mipspace does not email abuse and anyone can say an ip is bad and eventually they block the entire range. At this time only a small % of our ip space is effected, nyc1 is the only location we offer that has less than 10% of all ip space on this list.. Personally, I would move to nyc2 if possible and reconsider who you email and what you email. They don't even follow quarantine removal processes. We're sorry we cannot accommodate your needs but will be more than happy to refer you to services that can relay to almost everyone in the world. I'm sorry, but we are unable to provide SWIP or rwhois for individual IPs.
Obviously this presents a challenge to running any sort of outgoing mail via a Digital Ocean droplet. From a rational economic perspective it is understandable as to why Digital Ocean wouldn't want to expend resources to deal with this: they are a bare-bones, low-cost hosting service, and MIPSpace are not behaving well. MIPSpace is no doubt fully aware that cloud services do not provide the service that they ask for, just as Digital Ocean is fully aware that the situation emerges because they are not sufficiently reducing the abuse of their service by spammers. Amazon EC2, for instance, requires some action and waiting on the part of a customer in order to permit outgoing email to pass, and no doubt Amazon has a much more developed internal apparatus for kicking bad customers and ensuring that their IP address space doesn't become polluted.
On EC2, however, you pay three times as much for the same processing power, which is possibly a case of getting what you pay for here. Equally I could shift to using MailChimp or similar, which absolutely solves the deliverability issues, but has its own setup costs in addition to the ongoing financial ones.
It is fortunate that MIPSpace is not widely used by email service providers, as the same situation that is occurring with Digital Ocean is no doubt playing out with out the other younger and cheaper cloud services. Moving my mail server might be a short term solution, but equally may drop me into an even more polluted IP address range.
So What is Going On Here?
The most logical explanation is that Digital Ocean is a spam-sender's and attacker's playgroup. It is cheap and droplets can be used to immediately send out mail. Bad actors no doubt constantly generate new accounts and rent servers for their nefarious purposes, which leads Digital Ocean IP addresses to become increasingly polluted and worthless for any use involving email.
Digital Ocean has chosen not to invest in earnestly dealing with this while they are in growth mode, which may be the rational choice on their part given that running a mail server on their system is probably not a common use case in comparison to all the other ways in which one can use a cloud service. Given that they recently had a large influx of venture capital, one might hope that they turn some of that to managing the growing pollution of their IP address space, however. If you are in the business of renting IP addresses, you should also be in the business of ensuring that they are clean and unencumbered.