50 Resources on Content Moderation and Mis/Disinformation for Aspiring Trust & Safety Professionals
Trust & Safety professionals and those responsible for content moderation policies and practices are in the midst of dealing with some of the most consequential aspects of internet safety – mis/disinformation during war and conflict.
Russia’s invasion of Ukraine has necessitated an immediate and nimble response from social media companies — but, of course, these issues are not new. Ongoing conflicts in Ethiopia, Yemen, and Syria, and elsewhere around the world should motivate similar decisions made with comparable urgency.
In this post, we have gathered a variety of articles detailing current content moderation issues as well as broader trust & safety resources as a very basic primer for those interested in the trust & safety profession and in approaches to content moderation – a starting point for those who are curious about some of the foundational scholarship and current best practices in the field.
The resources and articles that we have compiled here address an interconnected set of issues that trust & safety professionals are confronting and best practices for tackling the complex problems around content moderation, especially in the area of mis/disinformation.
At the bottom of the post, we’ve also included a number of relevant trust & safety job openings from the All Tech Is Human Responsible Tech Job Board.
What else should we add to this list of resources? Reach out and let us know!
Rebekah Tweed
rebekah/alltechishuman.org
LinkedIn | Twitter
Content Moderation during Russia’s invasion of Ukraine
On March 1, 2022, Harvard’s Shorenstein Center just released The Media Manipulation Casebook: Tracking Social Media Takedowns and Content Moderation During the 2022 Russian Invasion of Ukraine:
Tracking Social Media Takedowns and Content Moderation During the 2022 Russian Invasion of Ukraine | Media Manipulation Casebook (by April Glaser, Jazilah Salam)
Google blocked YouTube channels of Russian state-sponsored media outlets (in Europe):
Facebook and Twitter took down disinformation accounts featuring profile pics generated by AI:
Facebook, Twitter remove disinformation accounts targeting Ukrainians (nbcnews.com) (by Ben Collins & Jo Ling Kent)
Twitter added "Russia state-affiliated media website" labels.
Commercial Content Moderation: Human & Automated
The challenges involved in carrying out commercial content moderation are detailed by scholars like Sarah T. Roberts (Behind the Screen) as well as Mary L. Gray (Ghost Work), who outline the harms brought upon the globalized workforce doing the often traumatic work of screening and removing offensive material and labeling data to train the artificial intelligence systems that are necessary for introducing layers of automation into the process.
The question of how and to what extent automated tools can reduce the burden on human content moderators is a subject of ongoing debate, as evidenced by the set of articles below — and even during All Tech Is Human’s recent Human Experience (HX) livestream in which Tami Bhaumik, VP of Civility & Partnerships at Roblox, and Avriel Epps-Darling, computational social scientist and Ph.D. candidate in Human Development at Harvard Graduate School of Education, addressed the topic.
New America: Everything in Moderation: The Limitations of Automated Tools in Content Moderation (Report)
Thomson Reuters Foundation: Online content moderation: Can AI help clean up social media? (by Umberto Bacchi)
Content moderation, AI, and the question of scale (by Tarleton Gillespie in Big Data & Society)
Machine Learning for Content Moderation — Introduction: An overview of machine learning systems for online content moderation (by Devin Soni in Towards Data Science)
Ofcom - Use of AI in Online Content Moderation (White Paper)
Resources:
Trust & Safety Professional Association (TSPA.org)
“The work of trust and safety professionals is critical to the health and safety of the internet and society at large. It is often challenging and difficult work as they address some of the most pressing and hardest multidisciplinary issues in online trust and safety. As a membership association, we bring together professionals in this field and provide professional development, networking opportunities, and educational resources.”
Trust & Safety curriculum: “To effectively safeguard their users from harmful content, these companies must grapple with incredibly complex issues, from child sexual exploitation to misinformation, while also responding to new and rapidly evolving threats. Policies and decisions by companies have real-world consequences, whether it is the safety of an individual or political repercussions for a nation; it is no wonder that civil society groups, governments, and regulatory bodies are increasingly scrutinizing the measures technology companies take to preserve users’ safety and privacy.”
Selected resource: Metaphors in Moderation
Oasis Consortium (oasisconsortium.com)
User Safety Standards: “We believe that actionable user safety standards are critical to building ethical brands and creating a sustainable digital future. Trust & Safety professionals can use these guidelines to conduct a self-assessment, validate their work, and secure needed resources.”
Making Safety Universal in Online Dating | Equal, Inclusive, Safe: On Grindr’s best practices for gender inclusive content moderation
Digital Trust & Safety Partnership (dtspartnership.org)
Digital Trust & Safety Partnership Best Practices Framework: DTSP regards the five overarching Commitments as representing the necessary steps taken by Practicing Companies to identify and address harmful content and conduct while preserving free expression and other rights, including internationally recognized human rights standards, as well as the social and economic value of digital services. The DTSP Best Practices Framework marks the first ever attempt to articulate current industry efforts to address online Content- and Conduct-Related Risks. The Practices Framework focuses on considerations in the development, governance, enforcement, improvement, and clear documentation of digital products and services. Over time this Framework will evolve as they grow in maturity and can be assessed in more standard ways, such as in other disciplines like security and privacy.
The Safe Framework: Tailoring a Proportionate Approach to Assessing Digital Trust & Safety
Article for World Economic Forum: This is why you need to know more about digital trust and safety
Internet Commission (inetco.org)
Evaluation Framework for Digital Responsibility: “Our EVALUATION FRAMEWORK FOR DIGITAL RESPONSIBILITY enables procedural accountability. It looks at how organisational cultures, systems and processes align to support corporate digital responsibility. It has a particular focus on internet safety, freedom of speech and the ways in which decisions are made in relation to content, contact, conduct online. The revised second edition (January 2022) IDENTIFIES 124 QUALITATIVE AND QUANTITATIVE INDICATORS of how commercial, safety, and freedom of expression issues are balanced.”
Electronic Frontier Foundation (EFF.org)
Who Has Your Back?: 2019 report on content moderation practices: “This year’s Who Has Your Back report examines major tech companies’ content moderation policies in the midst of massive government pressure to censor.”
Mozilla Foundation (foundation.mozilla.org)
Internet Health Report Hub: What can be done to make the internet healthier?: “The Internet Health Report combines research and stories in publications that explore what it means for the internet to be healthy.”
Integrity Institute (integrityinstitute.org)
Metrics and Transparency: understand the scale and cause of harms occurring on social media platforms
Partnership on AI
Misinformation interventions are common, divisive, and poorly understood (by EMILY SALTZ Partnership on AI, USA; SOUBHIK BARARI Department of Government, Harvard University, USA; CLAIRE LEIBOWICZ Partnership on AI, USA; CLAIRE WARDLE First Draft, USA) in HKS Misinformation Review
Manipulated Media Detection Requires More Than Tools: Community Insights on What’s Needed by Claire Leibowicz, Jonathan Stray, Emily Saltz
5 Urgent Considerations for the Automated Categorization of Manipulated Media by Emily Saltz (PAI), Pedro Noel (First Draft), Claire Leibowicz (PAI), Claire Wardle (First Draft), Sam Gregory (WITNESS)
Encounters with Visual Misinformation and Labels Across Platforms: An Interview and Diary Study to Inform Ecosystem Approaches to Misinformation Interventions (by EMILY SALTZ and CLAIRE LEIBOWICZ, The Partnership on AI; CLAIRE WARDLE, First Draft)
Fact-Checks, Info Hubs, and Shadow-Bans: A Landscape Review of Misinformation Interventions by Emily Saltz, Claire Leibowicz
It matters how platforms label manipulated media. Here are 12 principles designers should follow. by Emily Saltz, Tommy Shane, Claire Leibowicz, Victoria Kwan, Claire Wardle
Labeling Misinformation Isn’t Enough. Here’s What Platforms Need to Do Next. by Claire Leibowicz, Emily Saltz
Global Network of Internet and Society Research Centers (networkofcenters.net)
Data & Society — Media Manipulation & Disinformation (datasociety.net/)
Source Hacking: Media Manipulation in Practice: “Source Hacking details the techniques used by media manipulators to target journalists and other influential public figures to pick up falsehoods and unknowingly amplify them to the public.”
Disinformation Action Lab: “The Disinformation Action Lab (DAL) at Data & Society forges new approaches to address the complex dynamics underpinning the spread of propaganda and disinformation."
Data Craft Library: The Manipulation of Social Media Metadata (by Amelia Acker) “Data Craft analyzes how bad actors manipulate metadata to create effective disinformation campaigns and provides tips for researchers and technology companies trying to spot this “data craft.””
Data Voids Library: Where Missing Data Can Easily Be Exploited (by Michael Golebiewski, danah boyd) “Data Voids demonstrates how manipulators expose people to problematic content by exploiting search engine results.”
Markkula Center for Applied Ethics at Santa Clara University - (Internet Ethics)
The Santa Clara Principles - On Transparency and Accountability in Content Moderation
Content Moderation Remedies (by Eric Goldman, Co-Director, High Tech Law Institute, Santa Clara University): “if a user’s online content or actions violate the rules, what should happen next?”
Round-Up of Materials from High Tech Law Institute conference: “Content Moderation and Removal at Scale” (by Eric Goldman, Co-Director, High Tech Law Institute, Santa Clara University)
Misinformation and Violence (by Rohit Chopra): The link between misinformation and violence needs to be addressed by legacy media, new media platforms, civil society, and regulatory bodies.
Oxford Internet Institute (OII)
DemTech | Opinion: We need a global panel on fake news (by Philip Howard and Sheldon Himelfarb in The Times)
DemTech | Opinion: What’s stunning about the misinformation trend -- and how to fix it (by Sheldon Himelfarb and Philip Howard on CNN.com)
Center for Critical Internet Inquiry (UCLA) (C2i2)
Critical Internet Speaker Series - C2i2 Scholars’ Council Speaker Series
Augmenting Social Media Content Moderation - NSF Project
Center for an Informed Public (University of Washington) (CIP)
What can we do to avoid contributing to the ‘fog of war’ online during Russia’s invasion of Ukraine?
Community Signal podcast: The ways anti-vaccine activists and others try to avoid content moderation on social media platforms
CNN Reliable Sources interview: CIP’s Kate Starbird discusses ‘participatory disinformation’
New York Times interview: CIP’s Rachel E. Moran on misinformation shared via cat videos
Misinformation Escape Room: Loki's Loop (lokisloop.org)
Project: Rumor Permutations: Tracking and Unpacking Rumor Permutations to Understand Collective Sensemaking Online
Emma S. Spiro (PI), Kate Starbird (Co-PI)
Project: Misinformation: Detecting Misinformation Flows in Social Media Spaces During Crisis Events Kate Starbird (PI), Emma Spiro (Co-PI), Robert Mason (Co-PI)
How Do You Solve a Problem Like Misinformation? (by Ryan Calo,Chris Coward, Emma S. Spiro, Kate Starbird, and Jevin D. West; in Science Advances)
Center for Information, Technology, and Public Life (University of North Carolina) (CITAP)
Media Manipulation and Disinformation Online (by Becca Lewis and Alice Marwick published by Data & Society)
Why Do People Share Fake News? A Sociotechnical Model of Media Effects (by Alice Marwick in Georgetown Law Technology Review)
Critical Disinformation Studies syllabus
Podcast: Does Not Compute
Internet Policy Research Initiative (MIT)
Harvard Shorenstein Center on Media, Politics, & Public Policy (ShorensteinCenter.org)
The Media Manipulation Casebook: Tracking Social Media Takedowns and Content Moderation During the 2022 Russian Invasion of Ukraine
Disinformation research initiative: “The Shorenstein Center addresses the spread and impacts of misinformation, disinformation, and media manipulation through a range of research-based approaches. We study the causes of disinformation, how it spreads through online and offline channels, why people are susceptible to believing bad information, and successful strategies for mitigating its impact.”
HKS Misinformation Review: Content Moderation topic
Technology & Social Change Program: “Led by Dr. Joan Donovan (@BostonJoan), The Technology and Social Change Project (TaSC) explores media manipulation as a means to control public conversation, derail democracy, and disrupt society. TaSC conducts research, develops methods, and facilitates workshops for journalists, policy makers, technologists, and civil society organizations on how to detect, document, and debunk media manipulation campaigns.”
Harvard Berkman-Klein Center for Internet and Society (BKC)
BKC's Institute for Rebooting Social Media workshop: It Could Happen (T)here: Transnational Advocacy Strategies Around Social Media “Social media facilitates engagement across cultures and time zones, and its impacts are global. When those impacts are negative, the harm — like all harms — is felt locally, and personally. But while the specifics are local, the harms can often be mirrored by similar effects elsewhere. This has proven a challenge to research and advocacy, as enormous, U.S.-based social media companies have been slow to register their global roles and react to severe impacts.”
Research Project: Harmful Speech Online
Publication: Content and Conduct - HOW ENGLISH WIKIPEDIA MODERATES HARMFUL SPEECH (by Hilary Ross, Casey Tilton, Justin Clark, Rob Faris, Urs Gasser, Adam Holland)
ADL - Center for Technology and Society (adl.org)
“The mission of ADL’s Center for Technology and Society is to ask the question “How do we secure justice and fair treatment for all in a digital environment?””
Report: Online Hate and Harassment: The American Experience 2021 “According to the latest results from ADL’s annual survey of hate and harassment on social media, despite the seeming blitz of self-regulation from technology companies, the level of online hate and harassment reported by users barely shifted when compared to reports from a year ago.”
Report: Hate in Social VR New Challenges Ahead for the Next Generation of Social Media
Resource: The Online Hate Index “a set of machine learning classifiers that detect hate targeting marginalized groups on online platforms”
Resource: Online Hate Ecosystem Primer “In order to combat hate online, it’s important to understand the ecosystem in which it thrives. To that end, experts at ADL have developed diagrams that map out the web of connections between and among bad actors, targets, institutions, behaviors, causes, and outcomes.”
Library: ADL Social Pattern Library “The ADL Social Pattern Library is an anti-hate by design initiative providing a set of product recommendations for social media platforms to help combat online hate, disinformation and extremism. The library includes both visual and interactive examples for how social media platform designers might mitigate hate. This pattern library is intended as a resource for social media interaction designers.”
Online Hate Prevention Institute (OHPI)
Basic Online Safety Expectations “In October 2021 the Online Hate Prevention Institute provided a submission to the Australian Government’s consultation on the draft Online Safety (Basic Online Safety Expectations) Determination. Today, the new Ministerial Determination was announced and has come into effect.”
How to Guides “The Internet is still the Wild West of free speech. Anyone can become the target of hate. OHPI’s aim is to get technology companies and governments to recognize and take action against hate speech. While we work on that, here are some tips on how to remain safe on the Internet and avoid becoming the victim of online hate. We have also created step-by-step guides on how to report hateful content if you encounter it. Reporting such content to the platform holders is the first step towards getting it removed. The more reports they receive of hateful content, the more pressured they feel to remove it… so never shy away from reporting hate, as you never know when you might become the target of it.”
Trust & Safety Job Openings
Here are some currently available Trust & Safety JOBS:
Anti-Defamation League (ADL) - Associate Counsel, Technology Policy & Advocacy
Berkman Klein Center for Internet & Society - Program Director, Rebooting Social Media
Amazon Web Services (AWS) - Senior Trust & Safety Policy Manager
TikTok - Product Manager MBA Intern (Trust and Safety) - Summer 2022
Grindr - Trust & Safety Specialist
Roblox - Director of Product - Trust and Safety
Finally, here’s an optimistic thread on the good news — what’s worked this week in combatting misinformation:
