Huge failure due to
This is Tiffiniy Cheng, Holmes Wilson and Joshua Blount
at Fight for the Future, a project of Center for Rights, which in turn is a 501(c)4 nonprofit with a 501(c)3 application in the works. Tiffiniy and Holmes are the founders. Joshua is our developer. We’re a new organization, but we’re well-known for our efforts to stop SOPA & PIPA, the infamous website-blocking legislation.
Back in January we decided to move to a new service (Action Kit) for sending emails. During that process, we asked our former provider (Blue State Digital) for an export of subscribed users only. Instead, they gave us an export that included all users, and as a result, our last mailing went to every one of the 84,000 people who had unsubscribed from our list. This makes us feel sick. To everyone that happened to, we're so sorry. It will never happen again.
The same mistake also destroyed our reputation with the anti-spam community. In February, Spamhaus contacted us to say that harvested emails were getting on our list, and we worked with them to a resolution: unsubscribing all 33,000 emails from the petition that seemed to be the problem. After we dumped these emails, Spamhaus gave us the go ahead to continue using our list. We thought we were in the clear. But, the migration error not only made it so we emailed everyone who unsubscribed, but also put all 33,000 emails that we threw away back onto our list, which from Spamhaus’s view looked like us trashing our agreement. Worse, coming from a new server at our new provider, it looked to Spamhaus and anyone else on the anti-spam frontlines that we were switching IPs to avoid their blocks (or even worse, that we’d been kicked off Blue State Digital).
So to Spamhaus (their listing) the Mailsleaze blog (their post), and everyone who works on fighting spam: we’re equally sorry, and we’re ready to do whatever it takes to rebuild your trust. As a first step, we’re going to give you a detailed account so you can see what’s happened, and a list of things we’re doing right now to fix it. We’ve shown this document to Action Kit and BSD so that they could verify its accuracy, and all emails are included here with permission. These are the areas we think are important (we’ve included dates, as well as key emails):
- 1. Timeline: How our list grew.
- 2. Timeline: February Spamhaus Issue
- 3. Timeline: Migration Disaster
- 4. Steps we're taking immediately
- 5. Steps we're considering
Finally, Fight for the Future is new on the scene, but its founders aren't. We've built ethically-founded free software projects that people love (Miro, Amara, OpenCongress) and we've worked for and alongside some of the internet’s most trusted organizations (Holmes was a campaign manager at the Free Software Foundation, Joshua comes from Canonical and together we’ve worked closely with Mozilla, Wikimedia, the EFF, and Public Knowledge). We have no conceivable interest in harvesting emails or emailing unsubscribed users. Even if we did, we never would. The anti-spam fight is about giving individual users control of the internet. That's what we're fighting for too. We're committed to building something lasting, fighting for issues that many of you support. I hope we're worthy of your understanding, forgiveness, and most of all your help in finding fixes that let us continue our work.
During the fight to stop SOPA, our ability to send emails to a large list was crucial. We used our growing list of supporters to drive phone calls to the House and Senate at crucial moments and organized in-person meetings with Senators in 30 states. Without that sustained activity, organized using email, the two spectacular protests would have seemed much less substantial to Congress. If SOPA or similar legislation returns, email will let us notify the millions of people who took action, starting right where we left off.
We still don’t know how harvested emails ended up on our list, but we know we need to take aggressive steps to prevent it. We’ve already had a hint of underhanded attacks: on Nov 22, 2012 our domain registrar Dreamhost pulled the plug on all our domains in response to an unverified complaint, days after the first SOPA protest. Public Knowledge's site was hacked the same week when somebody inserted malicious code on their page, causing their site to be blocked by StopBadware.org. If anyone with a botnet can shut down our ability to reach supporters by contaminating our list with trap addresses, that could sink us, so we’re going to start using a confirmed opt-in process to ensure that new addresses we collect are from real people who want emails from us.
If you can help, if you have any thoughts, or if you think there's anything missing, please let us know: firstname.lastname@example.org Thanks for reading this far — we hope the information below is even more helpful.
— Tiffiniy, Holmes, Joshua
Our list grew at moments when several sites with huge traffic (and many more small and mid-sized sites) were sending people to our pages with prominent links or embedded “write congress” forms. All of our forms include a clear statement that we will contact people about future campaigns. The vast majority also block multiple submissions from the same IP. Until Spamhaus told us on Feb 10 that we were hitting their spamtrap addresses, we had no reason to doubt the integrity of our list.
- a. Oct 21, 2011
- We launch our first campaign, about a Senate bill that would have made posting streaming videos a felony: Free Bieber.
- b. Oct 24, 2011
- Link from a popular Youtube show drove massive traffic resulting in many signups and, within a few days, an endorsement of remix culture from Justin Bieber himself! List size on 10/25: 62,531
- c. Nov 16, 2011
- We organize the first anti-SOPA protest, American Censorship Day. Huge sites like Reddit, Tumblr, Mozilla, and 4chan participate, either linking to our “Write Congress” pages, or including our “Write Congress” widget on their site (See infographic and screenshots). The protest sets the messaging and tactics for the Jan 18 blackout. List size on 11/17: 263,599
- d. Nov 16, 2011 - Jan 16, 2012
- Many medium-sized sites continue linking to AmericanCensorship.org and our PIPA video, and using our widget for weeks. We run several actions driving phone calls and emails to Congress. We see a much smaller but steady number of new signups per day. List size on 1/13: 959,260 (12k per day since 11/16)
- e. Jan 18, 2012
- During the January 18 SOPA protests, 71,800 websites pointed their users to our contact Congress / signup page including huge sites like WordPress.com and Wordpress.org, Google.com, reddit.com, TheCHIVE.com and Pinterest.com. List size on 11/19: 1,731,917
- f. Nov 16 - Jan 18, 2012
- In the course of our anti-SOPA campaigns, we had 1,000,000+ Facebook likes on AmericanCensorship.org, ~240,000 Likes on fightforthefuture.org/pipa and ~180,000 likes on SopaStrike.com/strike.
- g. Feb 9, 2012
- On the day we first heard from Spamhaus, our list was 2,467,099. It had been increasing 36k per day since 1/18. Many smaller sites had left their links in place for days after Jan 18, so the list continued to grow.
On February 10 Spamhaus told our CRM provider Blue State Digital that our list was dirty, that we were hitting Spamhaus traps. They told us that the trap addresses we were hitting belonged to pages on listed on sopastrike.com. They also told us (via BSD) that they had confirmed cases of failures to unsubscribe. We didn’t know how trap addresses got on our list, but we agreed with BSD and Spamhaus on removing all emails from one sopastrike.com petition as a first step. Our understanding was that, if this didn’t solve the problem, we would remove a larger set of emails. BSD fixed a bug in their unsubscribe page, and Spamhaus gave us the okay to send more emails. By that point we were already in the process of migrating to Actionkit, so we didn’t send any more emails to the full list until April 6th. (Two other emails in late March went to tiny parts of the list).
- a. Jan 17, 2012
- We send an email to 1,010,000 on the eve of the SOPA blackout. No Spamhaus flags.
- b. Jan 18, 2012
- We send an email to 1,731,917 on the day of the SOPA blackout. No Spamhaus flags.
- c. Jan 20, 2012
- Email sent to 2,386,522. First and only email to have a request for donations. No Spamhaus flags.
- d. Jan 25, 2012
- FFTF sent an email titled “SOPA has become toxic” to 2,461,702. No Spamhaus flags.
- e. Feb 9, 2012 - 11:57am
- FFTF sent an email titled “Help the protests this Saturday” to 2,467,099 email addresses on our mailing list (5,397 additional addresses)
- f. Feb 9, 2012 - 10:31am
- Blue State Digital tells us that the “Help the protests this Saturday” email was flagged by BSD/Spamhaus. This is the first time we’ve heard from Spamhaus.
BSD notifies us that we hit Spamhaus traps. The “upload” issue they mention in this email isn’t relevant: we’ve only imported emails from our own forms, and only on a couple occasions (A mobilecommons call form and a “censor your site” rails app we made for americancensorship.org).
- a. Feb 10 10:31am
Subject: Spam Email Issue
From: Client Services Manager at Blue State Digital
Hi Holmes, Josh and Tiffiny,
It seems like we are running into some problems with your recent email blasts that are being sent to your entire list. The mailing that was sent yesterday forced one of our production mail serves on to Spamhaus's spam block list. This is happening because your email list is extremely dirty. Please take a look at our email upload policy as a reminder (http://tools.bluestatedigital.com/kb/nuts-and-bolts-entry/wp-email-list-upload-policy). You should also be aware that this status can threaten the hosting of your full site, because repeated tagging as spam from your domain might threaten your site.
Until the list can be cleaned up, we are will be routing all of your mailings through our QOS mailer, which is isolated from our production mailers. This will probably hurt your delivery rates slightly. We would be more than happy to discuss this with you over the phone and to try and figure out why your email blasts are being flagged.
Client Services Manager
- b. Feb 10, 2012 - 11:35am
- We get on the phone with BSD immediately. They say that Spamhaus told them emails were scraped from the pages of sites that participated in the SOPA blackout (pages listed at sopastrike.com). They also say that a single email address keeps getting mailed. We sent BSD and Spamhaus links to all of our forms, and we sent BSD data to help them track down the single failed unsubscribe issue.
- c. Feb 10, 2012 - afternoon
- On our second call with BSD and BSD's sysadmin (BSD’s main contact to Spamhaus), we discuss different steps to resolve the problem, and decide to remove all emails from the http://sopastrike.com/ ‘request a site to protest’ petition. BSD and Spamhaus agree that, even though we don’t know the source of the problem, we know it’s linked to the “request a site” petition on the sopastrike.com front page, and therefore that removing these emails makes sense as a first step. We clarify the step we’ve decided on for BSD:
From: Joshua Blount
Feb 10 5:12pm
Thanks for getting on the phone to help us resolve this. The list we talked about dropping has the following details
Name: SOPA Strike Requests
Unique Emails: 33,973
And again, we're approving your team to delete this from our contacts DB with the understanding that the offending addresses identified by Spamhaus could potentially be contained in that list. Your team will contact Spamhaus after removing the list mentioned above explaining that our end users were entering data into that list that might have included a Spamhaus trap email address.
- d. Feb 11, 2012
- BSD Sysadmin confirms that after we remove the 41k contacts from SOPA Strike Requests Spamhaus has agreed to lift their block:
This will happen by 12 noon on Monday 2/13/2012.
Once the purge is complete, i will update Spamhaus (They have agreed to lift the block on the mailer server in question). You will then be able to recommence any and all email campaigns.
- e. Feb 13, 2012
- BSD Sysadmin confirms removal of potentially bad emails and gives greenlight on emailing (though it turns out there is one remaining issue with an unsubscribe bug):
All cons associated with the SOPA Strike Requests signup form have been successfully unsubscribed
Mailings are fine to begin again
- f. Feb 14, 2012
- With information we provided, BSD says they solved a bug in their unsubscribe flow. This was the last remaining issue. BSD says we are good to send emails again. At this point, we (right or wrong) considered the Spamhaus issue resolved until we knew otheriwse. From BSD's Operations Team Manager:
Hi everybody -
The data we got from your unsubscribe table has proved very helpful. As you know, we're putting together a workaround for the GMail '.' issue. From the other addresses that reported unsubscribe failures, we think we've been led to the root cause of the failure that Spamhaus reports. We'll keep you posted on that progress.
"Here's the note we sent to spamhaus about it: "Just sending the final results on the failed unsubscribe (opt out) issue.
We made a change in March of last year that resulted in us suppressing and ignoring certain internal framework errors relating to the unsubscribe process. This sometimes resulted in us marking email addresses in client databases as successfully processed and unsubscribed when they had not been.
We have reverted the change so that if there is an error during the unsubscribe process, an alert is generated and those rows will be retried the next time the unsubscribe process is run.
This is issue has been identified and resolved"
In September 2011, after considering both Blue State Digital and Action Kit, we decided on BSD. But in November we started to have serious issues and began discussing migration to Action Kit, making the final decision to move in January. When we asked for an export of “subscribed users”, BSD gave us a table that included all users with no information on unsubscribes or bounces.
This mean that our next mailing--months later--went to every user who had ever been on our list, breaking our promise to Spamhaus and any user who had asked to be unsubscribed. In retrospect we should have verified that the data was what we asked for. But there’s no conceivable misunderstanding; an export that included all users and no unsubscribe data was, essentially, a timebomb. Action Kit didn’t mess up: we told them that the data was what we believed it to be. BSD has apologized to us, and can verify that the mistake was theirs.
- a. Jan 12, 2012
- Sent action kit signed contract so they could begin preparing for migration
- b. Feb 9, 2012 - 3:00pm
We request export data from BSD to begin migration process (full text of email)
Joshua Craig Blount, Feb 09 15:13 (EST):
Titled: Help backing up subscribed users?
Hi! When I attempt to backup all data on subscribed users (about 2.6 million at this point) the page seems to crash with a non-specific error.
Can I get a dump (CSV preferred) of this data? Is there a way to do this myself without making everything break?
- c. Feb 10, 2012 - 10:30am
- We hear from Spamhaus for the first time that we are hitting their spamtraps, and agree on a solution with Spamhaus.
- d. Feb 14, 2012
- With the solution implemented, we receive a greenlight to send more mailings, but we don’t send any more mailings until April 6.
- e. Feb 28, 2012
- We put campaigns on hold while the migration is underway. Our setup on BSD required building self-hosted pages. Because this was time intensive, we decided instead to focus on building a better setup with Actionkit rather than investing more time in improving our flow for creating static pages.
- f. Mar 3, 2012
- Actionkit, seeing no bounce or unsubscribe data, asks us if the data we've given them includes mailable users only. We say yes, because that's what we asked for, and because BSD giving us a table with unsubscribed users--and no information to distinguish them--is unthinkable.
- g. Mar 13, 2012
- Actionkit completes import of data.
- h. Mar 13, 2012
- We ask BSD for a list of any new subscribes and unsubscribes since the last export. (This did not include users from contaminated sopastrike.com list that they had already unsubscribed.) Once we get it, we give this data to Action Kit
- i. Mar 26 & 29, 2012
- We send email two emails to tiny parts of the list: users in Iowa, and European users for whom we have country data (not many). Neither list is large enough to set off red flags.
- j. Apr 4, 2012 & Apr 6, 2012
- We send email to partial then full list which (and we don’t know it!) includes every unsubscribed user, every bounce, and the 33.9k emails we agreed with Spamhaus to remove.
- k. Apr 5, 2012
- Patrick at Actionkit reports a high number of bounces, but says it’s best to let the email run.
- l. Apr 6, 2012
- Patrick at Actionkit tells us we’ve been flagged by Spamhaus again, hitting “about 50” of Spamhaus’s trap addresses, an unthinkably high number.
- m. Apr 9, 2012
- We discover that the data we imported from BSD included every user, including unsubscribes, bounces, and the 33.9k emails we agree with Spamhaus to remove. From Spamhaus’s point of view, we trashed our agreement and switched providers. They tell Actionkit they want us “turned off”.
- n. Apr 11, 2012
- BSD confirms that the export was their mistake, apologizes and informs Spamhaus and Actionkit. We start working on a document for Spamhaus, our users, and the broader anti-spam community on what happened (the document you’re reading now).
From: BSD's Director of Client Services
Date: April 11, 2012
I think what in part we are dealing with here is an issue of timing. Our backup database dump we sent you guys was seemingly assembled before the initial Spamhaus issue. You are correct though, in reviewing what happened, we did send constituents that had been marked as unsubbed.
This was a mistake on our side and I do apologize for that.
Ideally we would have sent you a refreshed full file at the end of the engagement. I am sorry that this aspect has caused you so much grief. If you are in communication with Spamhaus about this issue, feel free to include us in this chain so we can verify this.
New mailing data is now loaded in to dropbox with should make this all clearer.
Now that we know what happened with the migration error, there are some obvious steps we’re taking.
- Re-remove all unsubscribed users from our list (DONE)
- Re-remove all bounces from our list (DONE)
- Re-remove all 33.9k addresses from sopastrike.com/ from list as per agreement with Spamhaus. (DONE)
- Remove an even larger number of email addresses that came from sopastrike.com/, up to 75,000.
- In any future migration, verify with multiple tests that all unsubscribe requests are preserved (DOCUMENTED)
- Remove any addresses entered on forms that didn't block multiple submissions from the same IP (our call congress widget and a censor your site tool).
Finally, given this huge screw up in honoring unsubscribes, we’d like to be even more careful in the future about spotting and resolving failed unsubscribes. We can’t make any specific commitment to timing, but we will also be adding an issue tracking system to make sure that any reported instance of a failed unsubscribe stays on our radar until it’s diagnosed and fixed.
The hard part: we know we’re on our last chance with Spamhaus and others, and we still don’t know the cause of the problem. If we take the above steps and keep hitting spamtraps, we’re screwed. If the above steps work, but somebody with a botnet later submits known harvested emails to our list, we’re screwed. So before we start sending emails again, we need a better plan. If you have expertise in this space and you have any ideas, please be in touch.
Here’s what we’re considering for future signup forms to keep out fake submissions while keeping it easy to, say, contact Congress:
- Use Facebook, Google, OpenID API for signups.
- CAPTCHA for anyone entering email
- Confirmed opt-in for anyone entering email
- Mutliple chances to confirm opt-in
- Go through our data looking for patterns consistent with single users or botnets
- Confirmed opt-in for all users who haven't visited our pages from a link in an email
- Confirmed opt-in for all users who only signed up on one of our forms
- Confirmed opt-in, with multiple chances to confirm, for all existing users
Here’s what we’re considering for existing users, to weed out any addresses that shouldn’t be there:
Confirmed opt-in for all existing users is tricky. We promised everyone who’s on our list that we’d contact them if SOPA came back. We don’t want to boot them from the list just because they neglect to open or read a single one of our emails carefully. Hopefully there’s a way to do this right.
If you’ve taken the time to read this far, it probably means you care about us and the situation we’re in. Thanks. If you also have expertise or thoughts on how we can move forward, please be in touch. email@example.com