Enigma 2020

Fighting spam, phishing, and other forms of abuse on the internet is typically seen as a detection problem: find signals that will identify the bad guys and then use these signals to block them. In this talk, I argue that the most difficult part of fighting abuse is not detecting and blocking the bad guys—it's figuring out whether they're there in the first place. What's the "background level" of spam and fake accounts? How can we figure out what our detection systems are missing? Which abuse problem is the most important one to work on right now?

In this talk, I will show how good measurement of abuse unlocks both prioritization of work and analysis of impact. I will present several approaches that Facebook's integrity teams have used to measure and prioritize their problems, including user reports, human labeling, and automated labeling, and offer scenarios in which each of these should and shouldn't be used.

I will also introduce the "Abuse Uncertainty Principle" which says that you can use a metric for measurement or detection, but not both. The Uncertainty Principle implies that measurement is never a finished project, but I will offer strategies for ensuring that your metrics are good enough to inform key decisions. Armed with these tools, you can go back to your product and find out how much abuse it's attracting, how good you are at stopping it, and where you need to invest next.