Huge failure due to massive error.

We’re sorry. :-(

This is Tiffiniy Cheng, Holmes Wilson and Joshua Blount
at Fight for the Future, a project of Center for Rights, which in turn is a 501(c)4 nonprofit with a 501(c)3 application in the works. Tiffiniy and Holmes are the founders. Joshua is our developer. We’re a new organization, but we’re well-known for our efforts to stop SOPA & PIPA, the infamous website-blocking legislation.

Back in January we decided to move to a new service (Action Kit) for sending emails. During that process, we asked our former provider (Blue State Digital) for an export of subscribed users only. Instead, they gave us an export that included all users, and as a result, our last mailing went to every one of the 84,000 people who had unsubscribed from our list. This makes us feel sick. To everyone that happened to, we're so sorry. It will never happen again.

The same mistake also destroyed our reputation with the anti-spam community. In February, Spamhaus contacted us to say that harvested emails were getting on our list, and we worked with them to a resolution: unsubscribing all 33,000 emails from the petition that seemed to be the problem. After we dumped these emails, Spamhaus gave us the go ahead to continue using our list. We thought we were in the clear. But, the migration error not only made it so we emailed everyone who unsubscribed, but also put all 33,000 emails that we threw away back onto our list, which from Spamhaus’s view looked like us trashing our agreement. Worse, coming from a new server at our new provider, it looked to Spamhaus and anyone else on the anti-spam frontlines that we were switching IPs to avoid their blocks (or even worse, that we’d been kicked off Blue State Digital).

So to Spamhaus (their listing) the Mailsleaze blog (their post), and everyone who works on fighting spam: we’re equally sorry, and we’re ready to do whatever it takes to rebuild your trust. As a first step, we’re going to give you a detailed account so you can see what’s happened, and a list of things we’re doing right now to fix it. We’ve shown this document to Action Kit and BSD so that they could verify its accuracy, and all emails are included here with permission. These are the areas we think are important (we’ve included dates, as well as key emails):

Finally, Fight for the Future is new on the scene, but its founders aren't. We've built ethically-founded free software projects that people love (Miro, Amara, OpenCongress) and we've worked for and alongside some of the internet’s most trusted organizations (Holmes was a campaign manager at the Free Software Foundation, Joshua comes from Canonical and together we’ve worked closely with Mozilla, Wikimedia, the EFF, and Public Knowledge). We have no conceivable interest in harvesting emails or emailing unsubscribed users. Even if we did, we never would. The anti-spam fight is about giving individual users control of the internet. That's what we're fighting for too. We're committed to building something lasting, fighting for issues that many of you support. I hope we're worthy of your understanding, forgiveness, and most of all your help in finding fixes that let us continue our work.

During the fight to stop SOPA, our ability to send emails to a large list was crucial. We used our growing list of supporters to drive phone calls to the House and Senate at crucial moments and organized in-person meetings with Senators in 30 states. Without that sustained activity, organized using email, the two spectacular protests would have seemed much less substantial to Congress. If SOPA or similar legislation returns, email will let us notify the millions of people who took action, starting right where we left off.

We still don’t know how harvested emails ended up on our list, but we know we need to take aggressive steps to prevent it. We’ve already had a hint of underhanded attacks: on Nov 22, 2012 our domain registrar Dreamhost pulled the plug on all our domains in response to an unverified complaint, days after the first SOPA protest. Public Knowledge's site was hacked the same week when somebody inserted malicious code on their page, causing their site to be blocked by StopBadware.org. If anyone with a botnet can shut down our ability to reach supporters by contaminating our list with trap addresses, that could sink us, so we’re going to start using a confirmed opt-in process to ensure that new addresses we collect are from real people who want emails from us.

If you can help, if you have any thoughts, or if you think there's anything missing, please let us know: repairs@fightforthefuture.org Thanks for reading this far — we hope the information below is even more helpful.

— Tiffiniy, Holmes, Joshua

1. Timeline: How our list grew

Our list grew at moments when several sites with huge traffic (and many more small and mid-sized sites) were sending people to our pages with prominent links or embedded “write congress” forms. All of our forms include a clear statement that we will contact people about future campaigns. The vast majority also block multiple submissions from the same IP. Until Spamhaus told us on Feb 10 that we were hitting their spamtrap addresses, we had no reason to doubt the integrity of our list.

a. Oct 21, 2011
We launch our first campaign, about a Senate bill that would have made posting streaming videos a felony: Free Bieber.
b. Oct 24, 2011
Link from a popular Youtube show drove massive traffic resulting in many signups and, within a few days, an endorsement of remix culture from Justin Bieber himself! List size on 10/25: 62,531
c. Nov 16, 2011
We organize the first anti-SOPA protest, American Censorship Day. Huge sites like Reddit, Tumblr, Mozilla, and 4chan participate, either linking to our “Write Congress” pages, or including our “Write Congress” widget on their site (See infographic and screenshots). The protest sets the messaging and tactics for the Jan 18 blackout. List size on 11/17: 263,599
d. Nov 16, 2011 - Jan 16, 2012
Many medium-sized sites continue linking to AmericanCensorship.org and our PIPA video, and using our widget for weeks. We run several actions driving phone calls and emails to Congress. We see a much smaller but steady number of new signups per day. List size on 1/13: 959,260 (12k per day since 11/16)
e. Jan 18, 2012
During the January 18 SOPA protests, 71,800 websites pointed their users to our contact Congress / signup page including huge sites like WordPress.com and Wordpress.org, Google.com, reddit.com, TheCHIVE.com and Pinterest.com. List size on 11/19: 1,731,917
f. Nov 16 - Jan 18, 2012
In the course of our anti-SOPA campaigns, we had 1,000,000+ Facebook likes on AmericanCensorship.org, ~240,000 Likes on fightforthefuture.org/pipa and ~180,000 likes on SopaStrike.com/strike.
g. Feb 9, 2012
On the day we first heard from Spamhaus, our list was 2,467,099. It had been increasing 36k per day since 1/18. Many smaller sites had left their links in place for days after Jan 18, so the list continued to grow.

2. Timeline: Spamhaus Issue

On February 10 Spamhaus told our CRM provider Blue State Digital that our list was dirty, that we were hitting Spamhaus traps. They told us that the trap addresses we were hitting belonged to pages on listed on sopastrike.com. They also told us (via BSD) that they had confirmed cases of failures to unsubscribe. We didn’t know how trap addresses got on our list, but we agreed with BSD and Spamhaus on removing all emails from one sopastrike.com petition as a first step. Our understanding was that, if this didn’t solve the problem, we would remove a larger set of emails. BSD fixed a bug in their unsubscribe page, and Spamhaus gave us the okay to send more emails. By that point we were already in the process of migrating to Actionkit, so we didn’t send any more emails to the full list until April 6th. (Two other emails in late March went to tiny parts of the list).

a. Jan 17, 2012
We send an email to 1,010,000 on the eve of the SOPA blackout. No Spamhaus flags.
b. Jan 18, 2012
We send an email to 1,731,917 on the day of the SOPA blackout. No Spamhaus flags.
c. Jan 20, 2012
Email sent to 2,386,522. First and only email to have a request for donations. No Spamhaus flags.
d. Jan 25, 2012
FFTF sent an email titled “SOPA has become toxic” to 2,461,702. No Spamhaus flags.
e. Feb 9, 2012 - 11:57am
FFTF sent an email titled “Help the protests this Saturday” to 2,467,099 email addresses on our mailing list (5,397 additional addresses)
f. Feb 9, 2012 - 10:31am
Blue State Digital tells us that the “Help the protests this Saturday” email was flagged by BSD/Spamhaus. This is the first time we’ve heard from Spamhaus.

Email chain:

BSD notifies us that we hit Spamhaus traps. The “upload” issue they mention in this email isn’t relevant: we’ve only imported emails from our own forms, and only on a couple occasions (A mobilecommons call form and a “censor your site” rails app we made for americancensorship.org).

a. Feb 10 10:31am
b. Feb 10, 2012 - 11:35am
We get on the phone with BSD immediately. They say that Spamhaus told them emails were scraped from the pages of sites that participated in the SOPA blackout (pages listed at sopastrike.com). They also say that a single email address keeps getting mailed. We sent BSD and Spamhaus links to all of our forms, and we sent BSD data to help them track down the single failed unsubscribe issue.
c. Feb 10, 2012 - afternoon
On our second call with BSD and BSD's sysadmin (BSD’s main contact to Spamhaus), we discuss different steps to resolve the problem, and decide to remove all emails from the http://sopastrike.com/ ‘request a site to protest’ petition. BSD and Spamhaus agree that, even though we don’t know the source of the problem, we know it’s linked to the “request a site” petition on the sopastrike.com front page, and therefore that removing these emails makes sense as a first step. We clarify the step we’ve decided on for BSD:
d. Feb 11, 2012
BSD Sysadmin confirms that after we remove the 41k contacts from SOPA Strike Requests Spamhaus has agreed to lift their block:
e. Feb 13, 2012
BSD Sysadmin confirms removal of potentially bad emails and gives greenlight on emailing (though it turns out there is one remaining issue with an unsubscribe bug):
f. Feb 14, 2012
With information we provided, BSD says they solved a bug in their unsubscribe flow. This was the last remaining issue. BSD says we are good to send emails again. At this point, we (right or wrong) considered the Spamhaus issue resolved until we knew otheriwse. From BSD's Operations Team Manager: (BSD later gave us more information on what the unsubscribe problem was, in an email on April 11)

3. Timeline: Migration Disaster

In September 2011, after considering both Blue State Digital and Action Kit, we decided on BSD. But in November we started to have serious issues and began discussing migration to Action Kit, making the final decision to move in January. When we asked for an export of “subscribed users”, BSD gave us a table that included all users with no information on unsubscribes or bounces.

This mean that our next mailing--months later--went to every user who had ever been on our list, breaking our promise to Spamhaus and any user who had asked to be unsubscribed. In retrospect we should have verified that the data was what we asked for. But there’s no conceivable misunderstanding; an export that included all users and no unsubscribe data was, essentially, a timebomb. Action Kit didn’t mess up: we told them that the data was what we believed it to be. BSD has apologized to us, and can verify that the mistake was theirs.

a. Jan 12, 2012
Sent action kit signed contract so they could begin preparing for migration
b. Feb 9, 2012 - 3:00pm
We request export data from BSD to begin migration process (full text of email)
c. Feb 10, 2012 - 10:30am
We hear from Spamhaus for the first time that we are hitting their spamtraps, and agree on a solution with Spamhaus.
d. Feb 14, 2012
With the solution implemented, we receive a greenlight to send more mailings, but we don’t send any more mailings until April 6.
e. Feb 28, 2012
We put campaigns on hold while the migration is underway. Our setup on BSD required building self-hosted pages. Because this was time intensive, we decided instead to focus on building a better setup with Actionkit rather than investing more time in improving our flow for creating static pages.
f. Mar 3, 2012
Actionkit, seeing no bounce or unsubscribe data, asks us if the data we've given them includes mailable users only. We say yes, because that's what we asked for, and because BSD giving us a table with unsubscribed users--and no information to distinguish them--is unthinkable.
g. Mar 13, 2012
Actionkit completes import of data.
h. Mar 13, 2012
We ask BSD for a list of any new subscribes and unsubscribes since the last export. (This did not include users from contaminated sopastrike.com list that they had already unsubscribed.) Once we get it, we give this data to Action Kit
i. Mar 26 & 29, 2012
We send email two emails to tiny parts of the list: users in Iowa, and European users for whom we have country data (not many). Neither list is large enough to set off red flags.
j. Apr 4, 2012 & Apr 6, 2012
We send email to partial then full list which (and we don’t know it!) includes every unsubscribed user, every bounce, and the 33.9k emails we agreed with Spamhaus to remove.
k. Apr 5, 2012
Patrick at Actionkit reports a high number of bounces, but says it’s best to let the email run.
l. Apr 6, 2012
Patrick at Actionkit tells us we’ve been flagged by Spamhaus again, hitting “about 50” of Spamhaus’s trap addresses, an unthinkably high number.
m. Apr 9, 2012
We discover that the data we imported from BSD included every user, including unsubscribes, bounces, and the 33.9k emails we agree with Spamhaus to remove. From Spamhaus’s point of view, we trashed our agreement and switched providers. They tell Actionkit they want us “turned off”.
n. Apr 11, 2012
BSD confirms that the export was their mistake, apologizes and informs Spamhaus and Actionkit. We start working on a document for Spamhaus, our users, and the broader anti-spam community on what happened (the document you’re reading now).

4. Steps we're taking immediately

Now that we know what happened with the migration error, there are some obvious steps we’re taking.

  • Re-remove all unsubscribed users from our list (DONE)
  • Re-remove all bounces from our list (DONE)
  • Re-remove all 33.9k addresses from sopastrike.com/ from list as per agreement with Spamhaus. (DONE)
  • Remove an even larger number of email addresses that came from sopastrike.com/, up to 75,000.
  • In any future migration, verify with multiple tests that all unsubscribe requests are preserved (DOCUMENTED)
  • Remove any addresses entered on forms that didn't block multiple submissions from the same IP (our call congress widget and a censor your site tool).

Finally, given this huge screw up in honoring unsubscribes, we’d like to be even more careful in the future about spotting and resolving failed unsubscribes. We can’t make any specific commitment to timing, but we will also be adding an issue tracking system to make sure that any reported instance of a failed unsubscribe stays on our radar until it’s diagnosed and fixed.

5. Steps we're considering

The hard part: we know we’re on our last chance with Spamhaus and others, and we still don’t know the cause of the problem. If we take the above steps and keep hitting spamtraps, we’re screwed. If the above steps work, but somebody with a botnet later submits known harvested emails to our list, we’re screwed. So before we start sending emails again, we need a better plan. If you have expertise in this space and you have any ideas, please be in touch.

Here’s what we’re considering for future signup forms to keep out fake submissions while keeping it easy to, say, contact Congress:

  • Use Facebook, Google, OpenID API for signups.
  • CAPTCHA for anyone entering email
  • Confirmed opt-in for anyone entering email
  • Mutliple chances to confirm opt-in

Here’s what we’re considering for existing users, to weed out any addresses that shouldn’t be there:

  • Go through our data looking for patterns consistent with single users or botnets
  • Confirmed opt-in for all users who haven't visited our pages from a link in an email
  • Confirmed opt-in for all users who only signed up on one of our forms
  • Confirmed opt-in, with multiple chances to confirm, for all existing users

Confirmed opt-in for all existing users is tricky. We promised everyone who’s on our list that we’d contact them if SOPA came back. We don’t want to boot them from the list just because they neglect to open or read a single one of our emails carefully. Hopefully there’s a way to do this right.

If you’ve taken the time to read this far, it probably means you care about us and the situation we’re in. Thanks. If you also have expertise or thoughts on how we can move forward, please be in touch. repairs@fightforthefuture.org