The unappreciated role of intent in algorithmic moderation of abusive content on social media

---

Implications

Substantial interdisciplinary literature seeks to define, detect, measure, and model different types of abusive content online. Industry efforts to moderate abusive content at scale rely heavily on automated, algorithmic systems. These systems routinely fail (in terms of both false positives and false negatives). This paper explains why and proposes a series of potential improvements.

The basic argument is straightforward. Whether or not content is abusive, according to most platform policies and legal definitions, depends on the state of mind of a human being, usually the speaker.  Yet algorithmic systems that process content (e.g., the speaker’s words) are unable to reliably determine a person’s state of mind. More and different kinds of information are needed.

The term abuse spans a spectrum of harmful language, including generalized hate speech and specific offenses such as sexism and racism. Definitions of different types of abuse often intersect and lack precise boundaries.  Later, we consider various definitions and taxonomies proposed to describe abusive content.

Common to most definitions of digital abuse is some notion of intent (e.g., French et al., 2023; Hashemi, 2021; Molina et al., 2021; Vidgen & Derczynski, 2020). Intent is a subjective state of mind attributable to an actual person, typically the speaker, poster, or sharer. In cognitive science, ethics, law, and philosophy, intent is a contested and complicated concept. As Frischmann & Selinger (2018) explain:

Intention is a mental state that is part belief, part desire, and part value. My intention to do something—say to write th[is] explanatory text… or to eat an apple—entails (1) beliefs about the action, (2) desire to act, and (3) some sense of value attributable to the act (p. 364).

In pragmatic ethical and legal contexts, the focus often turns to evidence of intent. For example, a written signature is considered an objective manifestation that a person intends to enter into a contract (Frischmann & Vardi, 2024). Thus, by including intent in definitions of abusive content, the implicit challenge is divining subjective state of mind from evidence manifested in the content itself and the surrounding context.

While we focus on intent as a central element in how platforms define and moderate abuse, we do not claim that interpreting intent is the only or universally preferred approach. Some frameworks prioritize observable harms or outcomes, especially in contexts like mis- and disinformation (Mirza et al., 2023; Scheuerman et al., 2021). Our aim is not to displace these frameworks but to critically assess the feasibility and implications of relying on intent when it is embedded in platform policy and annotation practices.

Unsurprisingly, intent is exceedingly difficult to capture algorithmically through analysis of short text (Gao & Huang, 2017; MacAvaney et al., 2019; Wang et al., 2020). ToxicBERT (Hanu & Unitary team, 2020), for example, can label sentences as hateful or toxic but lacks the ability to interpret context in the input (MacAvaney et al., 2019; Wang et al., 2020). ToxicBERT flags the sentence “I’m going to kill you if you leave the dishes for me again” as toxic and threatening; it fails to differentiate between literal and figurative language. Supervised algorithms like ToxicBERT rely on curated, typically human-annotated training datasets. However, the complexities of intent are eventually reduced to simple class labels—such as hate speech or not hate speech. The underlying assumption of this approach is that human annotators are furnished with adequate contextual information to make these judgements.

In this paper, we survey the landscape of the online abuse moderation policies of major social media platforms. We examine existing taxonomies of online abuse and survey existing algorithms for abuse detection. We focus on hate speech and bullying due to their prevalence, but our work can and should be extended to other content categories for which intent is relevant, including mis-/disinformation (Kruger et al., 2024). We examine the training datasets underlying these algorithms and the role (if any) of intent during dataset annotation. Finally, we survey the set of features extracted from training datasets and used for algorithm development. These features reflect the varied context available to algorithmic content moderation systems. Our findings motivate a set of recommendations to better align abuse detection algorithms with platform policies (see Figure 1).

Recommendation: Annotation—Introduce context and intent during dataset annotation

Dataset curators must recognize social, cultural, and other contextual variations present in natural language, and design annotation tasks sensitive to and mindful of these differences. Moreover, dataset curators must provide sufficient context to annotators to permit accurate assessment of intent, in particular (Anuchitanukul & Ive, 2022). Examples of such context include conversation threads between initiators and targets, user history and metadata, or norms defined by the specific platform. We propose a combined codebook-datasheet in the form of a structured set of questions for dataset curators and annotators to consider prior to annotation—particularly for online abuse datasets. This framework integrates annotation-specific guidance (codebook) with dataset-level context (datasheet). The full set of questions is in Appendix A. Responses to these questions should be made transparent and explicit to all stakeholders prior to annotation. The structure of this framework is informed by qualitative approaches that treat annotation as an interpretive task shaped by context and ambiguity, rather than a purely objective labeling (Charmaz, 2006). By encouraging dataset curators and annotators to reflect on definitions, contextual cues, and the relationships between initiators and targets, the Codebook-Datasheet promotes more consistent and intent-sensitive annotation practices.

Challenge: Trade-offs between capturing intent and achieving high annotation agreement

A primary challenge in enhancing the annotation phase with contextual details is the trade-off between accurately inferring an individual’s intent reflected through content and maintaining high agreement among annotators (Ross et al., 2017). Context can be subjective, and different annotators might interpret the same information differently based on their backgrounds, experiences, and biases (Joseph et al., 2017; Lynn et al., 2019). Providing more context also raises concerns about privacy, annotator fatigue, and task scalability. On the other hand, annotator disagreements are sometimes due to lack of sufficient context (Zhang et al., 2023). In cases where intent is ambiguous, omitting context may lead to systematic mislabeling or overly reductive interpretations of the content. Consequently, establishing standardized guidelines that incorporate diverse perspectives is essential to mitigate this concern and improve annotation consistency.

Recommendation: Detection—Develop context-aware, policy-aware, explainable models

Incorporation of contextual features into detection algorithms can substantially improve the accuracy of intent-based abuse detection (Markov & Daelemans, 2022; Menini et al., 2021). Recent progress in NLP, such as the introduction of retrieval-augmented generation (RAG) for large language models (Li et al., 2024), supports access to relevant contextual information from knowledge bases—for example, a user’s past interactions—in real-time (Shi et al., 2024). Unlike earlier methods that rely on fixed input windows or static embeddings of prior conversations, RAG retrieves the most semantically relevant context dynamically, enabling more targeted and interpretable use of prior information. This may be especially beneficial for platforms aiming to implement context-aware moderation policies that adapt to dynamic social norms. Likewise, state-of-the-art large language models (LLMs) are more effective at executing rule-based moderation (Kumar et al., 2023); platform policies can be retrieved as context and used directly during the algorithmic decision process. Importantly, advances in explainable AI (XAI) should help users and developers understand model outputs (Islam et al., 2023; Kohli & Devi, 2023). Examples of XAI techniques include LIME, SHAP, and attention heatmaps, which help explain why a model flagged a post as abusive (Muhammadiah et al., 2025). These tools can help moderators identify whether misclassifications stem from sarcasm, missing context, or ambiguous language—factors especially relevant when intent is at the core. XAI can be used, for example, to pinpoint why a model might misunderstand particular intentions or contexts and ensure compliance with social media moderation policies that require explanations of AI-driven decisions. We advocate for the implementation of policy-aware and dynamic abuse detection, which can be facilitated by XAI and retrieval-augmented models.

Challenge: Balancing performance and applicability

Incorporating contextual features presents a significant challenge, particularly in terms of performance, as standard detection models often prove to be overly optimistic (Menini et al., 2021). Ensuring that the system achieves high performance without compromising its applicability can be difficult; thus, advanced machine learning models and continuous algorithm training with updated datasets are required to address this balance effectively (Scheuerman et al., 2021). Additionally, integrating XAI methods improves transparency, but they may also reduce model efficiency when requiring simplification of underlying architectures (Crook et al., 2023). Thus, designing explainability into systems without sacrificing model performance remains an ongoing area of research.

Recommendation: Moderation—Incorporate wisdom of the crowd and establish feedback loops

While algorithmic systems play an essential role in content moderation at scale, they also have limitations. Algorithms struggle to process and make sense of the nuance and subtlety of human communication (Bender & Koller, 2020). Humans, although generally better at understanding context and intent, can exhibit inconsistency and bias (Basile et al., 2022).

We suggest that moderation can be improved by hybrid approaches, leveraging “wisdom of the crowd”—utilizing user reports and community feedback—alongside, and even integrated with, algorithmic systems to identify and moderate abusive content (Allen et al., 2021; Pröllochs, 2022). The potential effectiveness of crowd-based moderation is evident in several real-world implementations, such as Twitter’s Birdwatch (now X’s Community Notes), where users collaboratively add contextual notes to potentially harmful or misleading content;¹See: https://blog.x.com/en_us/topics/product/2021/introducing-birdwatch-a-community-based-approach-to-misinformation; https://help.x.com/en/using-x/community-notes Reddit’s voting and reporting mechanisms that shape content visibility;² See: https://support.reddithelp.com/hc/en-us/articles/7419626610708-What-are-upvotes-and-downvotes; https://support.reddithelp.com/hc/en-us/articles/360058309512-How-do-I-report-a-post-or-comment and Wikipedia’s multi-layered editorial and review workflows.³See: https://en.wikipedia.org/wiki/Wikipedia:Editing_policy Additionally, we recommend establishing clear pathways for feedback so that moderators can provide insights back to the automated detection systems to support iterative improvements, for example, as new labeled training data. Throughout these processes, platforms must design moderation systems to be sensitive to evolving definitions of inappropriate content and dynamic societal norms across regions and cultures.

Challenge: Managing the scale and the biases inherent in crowd-sourced moderation

The challenge in this phase lies in managing the scale of data and potential biases that can arise from crowd-sourced inputs. User reports can be influenced by personal biases or coordinated attacks, leading to false positives or negatives (Jhaver et al., 2019). Uneven participation rates mean that moderation decisions may disproportionately reflect the views of a narrow group of users. Moreover, the quality of crowd input may vary widely depending on task design, interface clarity, and community norms (Gadiraju et al., 2015). Implementing robust filtering algorithms to verify and validate user-generated reports before they influence the moderation process is crucial for maintaining the integrity of the system.

Recommendation: Design—Introduce friction to generate evidence of intent and enable more intentional actions

For the most part, we focus on the detection and evaluation of content flowing through social media systems as if the moderation pipeline is designed to operate independently, only triggering governance procedures upon detection of content that violates a policy. Of course, the actions of users depend to some degree on the affordances of the social media platform itself.

If we relax the assumption of independence between platform design and content moderation pipeline design, a range of friction-in-design (Frischmann & Selinger, 2018; Frischmann & Benesch, 2023) measures might generate reliable evidence of intent to better guide content moderation systems. For example, platforms might introduce prompts that query users about their intentions when users post or share (certain types of) content. Such prompts might be triggered based on different criteria, such as accuracy, authenticity, or intended audience. The prompt could provide a simple means for the user to express their intentions. The prompt might be framed in terms of purpose. Not only would a response provide potentially reliable evidence of intent that would be useful for moderation, but it also would provide the user with an opportunity to think about their own intentions. Other friction-in-design measures could provide users with knowledge about the potential consequences of their actions. Such measures would generate another source of reliable evidence about intent.

Prior research has shown that friction-based interventions—such as prompts during content creation, interstitial warnings, limits on message forwarding, and credibility nudges—can reduce misinformation sharing, offensive language, and improve user reflection (Clayton et al., 2020; Egelman et al., 2008; Fazio, 2020; Jahn et al., 2023; Kaiser et al., 2021). These examples underscore that friction can support both behavioral change and the elicitation of user intent for moderation purposes.

Challenge: Aligning business models and regulatory frameworks with slow governance for a safe, healthy digital environment

The basic idea of the friction-in-design strategy is to focus on the dynamic interactions between the social media platform and the content moderation pipeline from a design perspective. The associated challenges here are substantively different from the technical challenges we have outlined in prior sections. The success of friction-in-design approaches center around platform business models and regulatory frameworks that prioritize and reward healthy information ecosystems (Frischmann & Benesch, 2023). Frischmann & Sanfilippo (2023) emphasize the need to replace the dominant platform design logic that prioritizes frictionless, seamless interactions with one that embraces slow governance. Here, what that means is utilizing digital speedbumps not only to slow traffic but also, more importantly, for the instrumental functions of generating reliable evidence of intent and affording users the opportunity to be more intentional in their online behavior. Yet, as is probably all too obvious, this presents a fundamental challenge to existing business models.

Findings

Finding 1: Existing taxonomies of digital abuse are diverse and often ambiguous, making consistent categorization difficult.

Extensive literature details the diverse forms of online harm across various digital platforms (e.g., Arora et al., 2023; Im et al., 2022; Keipi et al., 2016; Scheuerman et al., 2021). These studies categorize a range of abusive speech, with prominent themes including hate speech, cyberbullying, and discrimination (Arora et al., 2023; Ghosh et al., 2024; Scheuerman et al., 2021). Each of these harms is consequent to nuanced user interactions, which are often platform-specific, culture-sensitive, and context-dependent. Prior work has highlighted the differing definitions of online harms in the literature (Fortuna & Nunes, 2018); this ambiguity poses challenges for the development of consistent and effective moderation pipelines.

Various approaches to classifying online abuse emphasize different parameters. Some taxonomies highlight the target of abuse—whether directed at individuals, groups, or concepts (Al Mazari, 2013; Vidgen et al., 2019; Waseem et al., 2017). Others focus on characteristics of the abuse—whether it is explicit or implicit (Mladenović et al., 2021), and still others explore subcategories of abuse or harm (Gashroo & Mehrotra, 2022; Lewandowska-Tomaszczyk et al., 2023). An overview of existing taxonomies is provided in Appendix B. Our work focuses on intent, which features prominently in platform policies but is often implicit in current taxonomies. For example, intent may be operationalized through the notion of targeting. We emphasize two common types of explicit online abuse: hate speech and cyberbullying (Wiegand et al., 2019). These forms of abuse have not only received significant attention in natural language processing (NLP) but reflect intent—deliberate aim to harm or intimidate specific individuals or groups.  To clarify how these categories are typically defined in the literature, we provide the following commonly used definitions:

• Hate speech: speech that attacks or discriminates against a person or group on the basis of attributes such as race, religion, ethnic origin, national origin, sex, disability, sexual orientation, or gender identity (Brown, 2017; Lepoutre et al., 2023)
• Cyberbullying: the use of electronic communication technologies like the internet, social media, and mobile phones to intentionally harass, threaten, humiliate, or target another person or group (Campbell & Bauman, 2018; Wright, 2021)

Finding 2: Platform moderation policies invoke user intent, yet intent is difficult to observe directly.

Intentions are a state of mind. People exercise their autonomy by acting upon their intentions. Most often, we rely on external manifestations, such as what people say or do (Frischmann & Selinger, 2018) to learn another person’s intent. In content moderation, like other contexts, intent is thus inexorably intertwined with actions. When attempting to determine the intent associated with abusive content, one must identify the relevant actor(s) and action(s). Notably, content is a sociotechnical artifact with history and context, and multiple relevant actors and actions may be involved with its arrival in the platform. Major platforms like Twitter (now “X”) and Facebook (now “Meta”) emphasize the importance of intent in content moderation. For example, Twitter’s guidelines state (emphasis added):

Violent entities are those that deliberately target humans or essential infrastructure with physical violence and/or violent rhetoric as a means to further their cause.

Hateful entities are those that have systematically and intentionally promoted, supported and/or advocated for hateful conduct, which includes promoting violence or engaging in targeted harassment towards a protected category.⁴See: https://help.Twitter.com/en/rules-and-policies/violent-entities (accessed May 2024).

Instagram also addresses the complexity of moderating hate speech by considering the intent associated with the act of sharing. The platform allows content that might be deemed hateful if it is shared to challenge or raise awareness about the issues discussed, provided this intent is clearly communicated (emphasis added):

It’s never OK to encourage violence or attack anyone based on their race, ethnicity, national origin, sex, gender, gender identity, sexual orientation, religious affiliation, disabilities, or diseases. When hate speech is being shared to challenge it or to raise awareness, we may allow it. In those instances, we ask that you express your intent clearly.⁵See: https://help.instagram.com/477434105621119 (accessed May 2024).

TikTok describes hate speech as intentional as well (emphasis added):

( … ) content that intends to or does attack, threaten, incite violence against, or dehumanize an individual or group of individuals on the basis of protected attributes like race, religion, gender, gender identity, national origin, and more.⁶See: https://newsroom.tiktok.com/en-us/countering-hate-on-tiktok (accessed May 2024).

Finding 3: Many publicly available datasets of abusive content lack detailed annotation procedures and do not systematically incorporate intent or platform-specific context.

Our review highlights several key challenges to the alignment between platforms’ policies around abuse and the datasets used to train abuse detection algorithms:

• Ambiguity in definitions of digital abuse: Compounding the diversity of definitions and taxonomies of abuse proposed in academic work and discussed above, instructions to annotators are often vague. This can result in lack of reusable training data and benchmarks. In Figure 2, we provide excerpts describing annotation processes from surveyed papers. While we cannot determine whether more comprehensive information was provided to annotators due to limited reporting, excerpts alone reveal substantial variability in definitions and instructions.

• Inconsistent consideration of intent during annotation: Table 1 shows an excerpt from the dataset paper summary (see Appendix D for the full table).Of the dataset papers surveyed, 47.6% (20 of 42) explicitly mentioned intent during annotation, while 35.7% (15 of 42) provided context to annotators to help them better infer intent. In addition, 33.3% (14 of 42) of the papers required annotators to identify the target of abuse. Just three papers (Van Hee et al., 2018; Vidgen, Nguyen, et al., 2021; Ziems et al., 2020) provided contextual information to annotators, explicitly acknowledged the role of intent in annotation guidelines, and requested annotators to identify the target of the abuse.
• Cross-platform differences: Text and annotation instructions are typically provided to annotators outside of the platforms from which they were collected. Labels are considered universal, rather than tailored to the environments and operational settings of specific social media platforms.

Reference	Source	Author-defined scope	Content type	Context provided	Target annotation	Intent mentioned
(Waseem & Hovy, 2016)	Twitter	Hate Speech	Text	No	No	No
(Waseem, 2016)	Twitter	Hate Speech	Text	No	No	No
(Golbeck et al., 2017)	Twitter	Online Harassment	Text	No	No	Yes
(Chatzakou et al., 2017)	Twitter	Cyberbullying	Multimodal	Metadata	No	Yes
(Gao & Huang, 2017)	Fox News	Hate Speech	Text	Conversation	No	No
(Davidson et al., 2017)	Twitter	Hate Speech	Text	No	No	Yes
(Gao et al., 2017)	Twitter	Hate Speech	Text	No	No	No
(Jha & Mamidi, 2017)	Twitter	Sexism	Text	No	No	No
(Van Hee et al., 2018)	ASKfm	Cyberbullying	Text	Conversation	Yes	Yes
(Fersini et al., 2018)	Twitter	Misogyny	Text	No	Yes	Yes

Finding 4: Current detection algorithms often fail to account for contextual and dynamic aspects of hate speech and cyberbullying, limiting their ability to infer intent and generalize across settings.

We prompted GPT-3.5 to classify several statements as abusive or not abusive, both with and without contextual cues (see Figure 3). We used GPT-3.5 because it is a widely available, general-purpose language model that can respond to contextual prompts, allowing us to observe how variations in context and intent may affect classification outcomes. We observe the impact of context and intention on model interpretations of each statement. However, many existing models, like those based on BERT, fall short in capturing context to infer intent. We evaluated four different models (Barbieri et al., 2020; Das et al., 2022; Mathew et al., 2021; Vidgen, Thrush, et al., 2021) using the same examples as those tested with GPT-3.5. The results (see Figure 3) show inconsistent outputs and a significant reliance on word-level cues, failing to account for the underlying user intent that contextualizes meaning.

Our context analysis of the papers on online abuse detection algorithms confirmed that while detection models have made notable advances in identifying abusive content through analysis of textual data, they often fail to consider the complex nature of social media interactions, which include aspects like user status, social network structures, and offline context. Figure 4(A) shows the co-occurrence network of ten features extracted and used to capture context and infer intent. The size of each node reflects how many papers included the corresponding feature, while the width of each edge represents how often two features appeared together in the same paper.  We observe that user metadata are the most frequently considered features, with 16 (9.5%) algorithms using them for detection. Similarly, linguistic cues are used in 14 (8.3%) studies and post metadata in 10 (5.9%). In contrast, psychological and cognitive dimensions, and policy or rule-aware models are less frequently explored, with just four (2.2%) and two (1.1%) studies, respectively. A majority of reviewed papers, 58.9% (99 of 168), focus on comparing model performance and benchmarking against various metrics. Figure 4(B) shows a histogram of papers according to the number of intent-related features considered in a paper. Among 168 surveyed papers, three of them present models incorporating six out of the ten features listed above, meaningfully including context and potentially addressing the issue of intent (Dhingra et al., 2023; López-Vizcaíno et al., 2021; Ziems et al., 2020). Building upon Ziems et al. (2020), a subsequent paper utilized the dataset from the earlier study and applied methodological improvements (Dhingra et al., 2023).

Based on these observations and our review of the literature, we summarized the current limitations of detection algorithms for online abuse.

• Insufficient context for inferring intent: Because intent is rarely explicit in language, detecting it often requires additional context beyond the text itself. Traditional text-based models, though robust in their linguistic analyses, fall short in understanding the complexities of group dynamics in the spread of online abuse (Salminen et al., 2018). For instance, an abusive narrative might emerge and propagate not merely because of its textual content but due to the influence and endorsement of a closely-knit group within the network (Marwick & Lewis, 2017). These group-based interactions are influenced by, for example, shared ideologies, mutual affiliations, or even orchestrated campaigns, which sometimes employ subtle linguistic cues not easily detectable by conventional text-based models. The complexity is further compounded when they employ tactics like code-switching, euphemisms, or meme-based communication, thereby effectively circumventing text-based detection mechanisms.
• Static perspectives: Social norms are dynamic and influence users’ interpretation of what constitutes abusive content (Crandall et al., 2002). Traditional detection models often fail to account for these longitudinal shifts. For example, a phrase may take on a new meaning in the digital realm, and the rapid evolution of internet slang also necessitates a more flexible approach for continuous training and updating.
• Poor generalizability: Model performance can fluctuate significantly even when utilizing the same dataset, with optimal outcomes frequently exclusive to that specific dataset (Leo et al., 2023). These inconsistencies highlight the need for robust testing to understand the generalizability of existing methods in new contexts.

Methods

Online abuse datasets

We reviewed the documentation provided in the associated papers of existing online abuse detection datasets, focusing on any available descriptions of data sources, categories, modality, and annotation procedures. We categorize each based on the extent to which it considers the context and intent of abusive expressions during annotation. We reviewed papers available between 2016 and 2024 using Scopus search along with datasets referenced in Vidgen & Derczynski (2020).⁷See: https://hatespeechdata.com/ (accessed May 2024) and citation search. The dataset selection and review process involved two researchers. One researcher drafted the initial table, and the second independently reviewed the associated papers in detail to identify any discrepancies. No inconsistencies were observed during this process. Further details, search terms, inclusion criteria, and a PRISMA diagram for the screening pipeline are provided in Appendix C.

Appendix D outlines papers reviewed, noting whether any contextual information was provided to annotators, whether intent was explicitly mentioned in annotation instructions, and whether the annotation task included identification of a target person or group.

• Intent mentioned: We capture whether annotation guidelines mention intent, intention, etc., anywhere in instructions or definitions.
• Context provided: When provided, context takes one of two forms: 1) conversations (annotators are provided with conversation/text surrounding the text being annotated, enabling them to infer intent by understanding the broader exchange) and 2) metadata (annotators are provided with user-level metadata, for example, profile content, geographical location, or post-level metadata, for example, images or text extracted from images, which may offer additional clues about intent).
• Target annotation: Annotation guidelines request that the person or group targeted by a comment be identified during annotation.

Online abuse detection algorithms

We reviewed the papers that described and examined abuse detection models, categorized features utilized by these models, and identified the gaps that exist in effectively detecting abuse. We queried SCOPUS for papers presenting abuse detection algorithms published between 2013 and early 2024. Similar to our survey of datasets, specific search terms, inclusion criteria, and a PRISMA diagram for the screening pipeline are provided in Appendix C. The review and categorization were also conducted by two researchers who cross-checked decisions and engaged in discussion to ensure consistency. No inconsistencies were reported during this process.

We categorized the features used by detection models in our survey that go beyond traditional text-based methods such as bag-of-words, TF-IDF, and embeddings. These features fall into the following ten categories: user metadata, post metadata, image and video content, psychological and cognitive signals, conversational context, graph structure, policy- or rule-aware signals, sentiment, topical or thematic information, and linguistic cues. Appendix E provides a detailed breakdown and examples for each category, and Appendix F includes a full mapping of these features to the corresponding papers in our survey.