How alt-tech users evaluate search engines: Cause-advancing audits

---

Implications

A user on the White nationalist forum Stormfront created a thread titled “Which search engine is most conducive to advancing our cause?”.¹https://www.stormfront.org/forum/t1376072/ In another thread titled “What are non-Jewish alternatives to Google?”, a commenter shared their strategy of querying “goy”—a term White supremacists often use to refer to themselves—in multiple search engines “to determine if the search engine is following Jewish propaganda.”²https://www.stormfront.org/forum/t1245794/ The commenter then encouraged other users to perform web search using Ecosia or Yandex. A user on 4chan’s “Politically Incorrect” (/pol/) board recounted evaluating 10 different search engines on their ability to return anti-vaccine content for the query “vaccine injuries,” lamenting that Google “would just provide a bunch of fact-checker garbage results.”³https://archive.4plebs.org/pol/thread/419356626/#419361999 The user then praised Yandex and encouraged other users to search “Holocaust fake” in both Yandex and Google. We observed many users on Gab performing similar comparisons.

Each of these comments contains an informal version of a search engine audit—a type of study where researchers query a set of terms across one or more search engines and analyze the results—conducted on an alt-right platform (social networks favored by White supremacists and groups banned from traditional media). In traditional academic search engine audits, researchers often evaluate search engine result pages (SERPs) to answer questions about the accuracy, fairness, or impartiality of one or more search engines (e.g., Makhortykh et al., 2020; Urman et al., 2022).⁴The belief that systems should be guided by fairness, impartiality, and accuracy reflects a certain value system, and indeed, all algorithmic audits around social phenomenon can be viewed as activist and in furtherance of a cause (Metaxa et al., 2021). While alt-tech platform audits adopt certain elements of academic audits, like comparing search results across search engines for a set of queries, alt-tech audits tend to diverge in both goals and methods. Whereas academic audits follow systematic methodologies to understand biases in algorithms and the resulting implications for all users, alt-tech forum audits are informal, explicitly activist, and aim to identify and promote search engines that return ideologically aligned results (e.g., on 4chan’s /pol/ board, results containing racist, conspiratorial, or pseudoscientific content). We term these non-academic, often-informal evaluation and promotion strategies “cause-advancing” search engine audits.⁵While the audits are usually informal, some cause-advancing audits we observed approach standard academic methodologies, but with questionable query selection and evaluation metrics (e.g., Ohlan, 2024; Suede, 2022). This practice mirrors the dynamics of selective trust observed in fragmented media ecosystems, where users embrace sources that reinforce beliefs and biases and reject sources that challenge them.

The fragmentation of traditional and social media ecosystems has facilitated the maintaining of large-scale bespoke realities where users can select or manipulate information, experiences, or beliefs to align with personal preferences or biases. This process allows users to “construct or curate a version of reality—one that can be tailored to fit whatever desires or agendas drive [them]” (DiResta, 2024, p. 41). This is an important phenomenon in explaining populist expertise, the rejection of mainstream and scientific consensus in favor of “home-grown” knowledge (Marwick & Partin, 2022). Although these dynamics have extensively been studied in the context of both social and traditional media fragmentation (Donovan et al., 2019; Figenschou & Ihlebæk, 2019), the politicization and fragmentation of information retrieval—the primary function of search engines—has received far less attention (Tripodi, 2018). Just as users can curate social media feeds and print media diets that reinforce pre-existing beliefs, they can also curate web search experiences that reinforce a given belief system. Consequently, in this study, we often observed users on /pol/ promoting search engines based on their ability to return results that promote White nationalism, pseudoscience, conspiracy theories, or medical misinformation.

We found that Yandex, which was purchased in 2024 by a Russian investment fund with close ties to the Kremlin, was the most recommended search engine promoted in cause-advancing audits. Even before its purchase, recent studies have shown that Yandex has increasingly served as a tool for spreading propaganda and disinformation that aligns with Kremlin objectives (Daucé & Loveluck, 2021; Kravets et al., 2023), including overwhelmingly returning results claiming COVID-19 originated in the United States (Kravets & Toepfl, 2022). Prior research has also shown that Kremlin propaganda outlets attempt to manipulate their search rankings and that Yandex amplifies pro-Kremlin misinformation and propaganda (Kuznetsova et al., 2022; Williams & Carley, 2023). While we have only quantitatively evaluated /pol/ in this paper, we found that searching for “Yandex” on Gab, Truth Social, and even X immediately surfaces similar cause-advancing audits. On X, we observed Yandex-promoting cause-advancing audits pushed by conspiratorial accounts with hundreds of thousands of followers (e.g., The Flat Earther, 2024; The White Rabbit Podcast, 2024). Additionally, though we primarily focused on cause-advancing audits in alt-right (far right, White supremacist) communities, we note that informal audits can be conducted by users on other parts of the political spectrum. For example, we observed an account with the name “Use Yandex Search Engine for Anti-Zionist Searches” with over 85,000 followers that appropriates rhetoric from the pro-Palestinian movement to spread antisemitic conspiracies, including the promotion of The Protocols of Elders of Zion, the Great Replacement myth, and blood libel.⁶https://x.com/_NicoleNonya/status/1876310361193754991 Social media and alt-tech users that are exposed to these audits are likely at higher risk of unknowingly consuming Kremlin propaganda, which could become more extreme following Yandex’s recent change in ownership.

This finding is particularly concerning given the role of search engines in shaping behaviors and political decisions. Research has shown that search rankings can have a substantial impact on both click-through rate and political decision-making (Dean, 2022; Epstein & Robertson, 2015). Annual global surveys by Edelman since 2016 have consistently found that users trust search engines over both traditional and social media (Edelman Trust Management, 2019). Furthermore, exit-poll interviews following the 2017 Virginia gubernatorial election found that voters often used Google to research candidates, treating it as an unbiased intermediary for information, while often misinterpreting the meaning of its returned website rankings (Tripodi, 2018). Users being encouraged to seek out ideologically aligned search engines because of cause-advancing audits risks deepening political polarization, conferring undue legitimacy to misinformation, spreading conspiracies, and exposing users to state-backed propaganda.

More research and more public education on search engines are both needed. Recent academic search engine reliability audit studies have suggested that Google returns the most reliable English-language results on highly politicized topics of any search engine (Kuznetsova et al., 2022; Urman, Makhortykh, & Ulloa, 2022; Urman, Makhortykh, Ulloa, & Kulshrestha, 2022). Large-scale audits have also found little search bias in Google across political lines (Metaxa et al., 2019; Robertson et al., 2018). If Google Search were to be broken up as a consequence of new or ongoing anti-trust litigation against the company (McCabe, 2024), smaller search engines could receive more visibility. Further research is needed to understand how a decrease in Google’s market share could impact the dissemination of unreliable information to the public. As previous research has shown that search rankings can have substantial impacts on user beliefs (Epstein & Robertson, 2015; Zweig, 2017), more needs to be done to inform the public of the limitations and biases built into information retrieval systems. We call for increased attention from policymakers and academics to the accuracy, biases, and impacts of smaller search engines. Media and digital literacy education programs in as early as middle school should teach algorithmic audits to help students critically evaluate information retrieval systems alongside other sources.

Findings

Finding 1: Cause-advancing audits are regularly conducted on 4chan’s /pol/ board.

After pulling 250,000 mentions of 30 different search engines from 4chan’s “Politically Incorrect” /pol/ board, we identified 1,218 comments that both mentioned at least three search engines and recommended one or more search engines. Comments meeting these criteria appeared an average of 15 times per month between January 2018 and July 2024. Figure 1 displays the monthly mentions of each search engine in the 1,218 /pol/ comments in this same time period. We have noted that spikes in comparisons occurred at the onset of the COVID-19 lockdowns in 2020 and during the vaccine rollout in the spring of 2021. Much of the 2020 spike was driven by a thread titled “The Great Jewtube Exodus,” which was reposted 45 times between March 7 and March 23, 2020.⁷Each thread contains a different thread ID. In each reposting, the author encouraged others to use Yandex, Bing, Startpage, or Brave Search instead of Google. These threads repeatedly resulted in conversations about search engine quality in the comments.

Finding 2: We observed that cause-advancing audits were evaluated on racial, ideological, or pseudo-scientific metrics.

Cause-advancing audits are inherently ideologically driven and evaluated. In this section, we explore several approaches that we qualitatively observed users employ when evaluating cause-advancing audits, along with specific keywords used. Many comments included no reasoning or rationale and simply promoted one or more search engines, while others offered reasoning. We identified four often-overlapping evaluation strategies: racial counting, specific URL retrieval, engine maligning, and Search Engine Result Page (SERP) ideological agreement. Quantifying and further codifying the usage of these strategies could be a fruitful path for future research.

Racial counting is an image-based evaluation strategy. In this type of evaluation, users search a term—for example, “American inventors,”—in multiple search engines and count the number of non-White individuals in the returned images. The engine that returns the fewest non-White results is then promoted by the commenter. We observed users conducting audit evaluations with this strategy for the queries “White American family,” “European art,” “European people art,” “White woman,” “White couple,” “blond women porn,” and others. The inverse scenario was also used; we observed several instances of users querying racial slurs and favoring the search engines that returned the most offensive racial caricatures in image results.

Specific URL retrieval involves a user searching a specific website, article, or video (sometimes pornographic) and then discussing the experience of attempting to find the URL in multiple different search engines. The evaluation is largely binary—either the URL is found or it isn’t, but how highly it ranks is often also considered. We observed comments using this strategy for queries like “Byrne vs Clinton Foundation RICO court,” “Christchurch manifesto,” “Dontell Jackson we thought they were White,” “site:bitchute.com Hitler,” and “The original great replacement manuscript.” Some users conducting searches in this category offered more general search tips, such as using “site:” or “AROUND” operators.

Engine maligning involves disparaging a search engine—most often without clear evidence or reasoning—and using this as justification to switch to alternative engines. These are often antisemitic in nature, and users on /pol/ frequently argue that Google and DuckDuckGo are owned by Jewish interests or people. The phrase “Jewgle” appeared in 42 unique comments. Comments also veered into homophobia, with one stating that Google is “the gay nightclub of the internet[;] the only things you find there are gay and shitty.” While some comments touched on privacy concerns, these were also often steeped in antisemitism: one comment recommending the privacy-centered search engines Searx and Qwant reminded users that there is “no excuse to be feeding a Jew your search history.” Most comments that we read did not outline specific events or issues justifying ire towards the maligned search engines.

SERP ideological agreement involves quantifying how many results align with the user’s ideology. Some alt-tech media users construct formal evaluation metrics and queries. There were several comments on /pol/ that explicitly used COVID-related terms to conduct these ideological agreement audits with queries like “Ivermectin sellers,” “MRNA gene therapy,” “Pfizer vaccine,” and “Covid vax side effects.” A Gab user performed a thorough search engine comparison on Substack to find the “least censored” search engine, evaluating search engines on their ability to return proof of election fraud in the United States, Russian state media sources, anti-vaccine content, among other topics; the author found Yandex was the “winner” when evaluating on these metrics (Suede, 2022). Some users on /pol/ also approach this from the opposite direction—using the presence of Anti-Defamation League or fact-checker URLs in top results as an indicator of poor search engine quality. One commenter angrily complained that top results for “pizzagate” are all “f*ing artificial f*ing ‘debunking’ bs” and recommended that others “use Bing or DuckDuckGo for real results.”

Finding 3: Yandex “wins” the most cause-advancing audits.

We used GPT-4o-mini to extract the relative rankings of search engines in each of the 1,218 comments that compared three or more search engines.  These were explicit endorsements or criticisms contained in a single comment. We found that Yandex was recommended as the top choice more than any other search engine (Table 1). DuckDuckGo and Google received the most criticism, much of which stemmed from antisemitic complaints surrounding their ownership. We included a third column in Table 1 containing the difference between the number of times a search engine was top-ranked and the number of times it was bottom-ranked. Yandex won on this metric as well.

Search Engine	#Comments Ranked as Best	#Comments Ranked as Worst	Best-Worst Difference
Yandex	322	213	109
DuckDuckGo	259	324	-65
Searx	226	181	45
Startpage	223	223	0
Bing	171	235	-64
Qwant	159	149	10
Google	139	298	-159

Methods

We iteratively extracted all mentions of 30 different search engines⁸https://web.archive.org/web/20240822142927/https://en.wikipedia.org/wiki/List_of_search_engines using the fouRplebsAPI (Buehling, 2022). Before using the API, we slightly modified several of the search engine names in the source Wikipedia list, for example, “Yahoo! Search” to“Yahoo.” All files and operations discussed in this section can be found with the released code and data. This collection process resulted in 250,000 comments or posts mentioning at least one search engine. The data contained substantial noise, so following minor cleaning operations, we elected to only explore the set of comments mentioning at least three search engines, as these very often contained content relevant to our research questions. This resulted in a set of 1,244 unique comments between January 2018 and July 2024.

Through OpenAI’s API we provided a prompt containing instructions as well as three comments and their associated annotations (i.e., few-shot learning) to GPT-4o-mini (GPT-4) and asked GPT-4 to annotate each additional comment. GPT-4 was instructed to extract 1) whether a user was recommending one or more search engines, 2) what relative ranking (within the comment) the user assigned to each search engine—where ties should receive identical ranks, and 3) any queries that the user claims to have searched in one or more search engine. Three example comment and annotation pairs were provided with the prompt, which can all be found in the accompanying code.⁹https://doi.org/10.7910/DVN/8RH7D2 As we had qualitatively observed beforehand, GPT-4 classified most of these comments (1,177) as engine comparisons. As some of these comments were re-posted on multiple threads, our final dataset contained 1,218 comments. All results presented in this paper were conducted on that set of 1,218 comments. User evaluation approaches were determined qualitatively after reading over 400 randomly sampled comments.

To validate GPT-4’s annotations, two human annotators independently annotated a random sample of 51 of the 1,244 considered unique comments. On task 1, identifying whether a comment was an engine comparison, all annotators’ labels aligned on 42 of the 51 comments (Fleiss’ Kappa = 0.40), with 40 of the comments being agreed-upon engine comparisons. However, much of the disagreement came from human annotators; there were only two cases where the human annotators agreed and GPT4 disagreed (Fleiss’ Kappa = 0.70). We discuss prompt creation and annotation challenges in more depth in the Appendix.

Task 2 was the most challenging of the three tasks, as it involved extracting the mentioned search engines and expressing their relative rankings. Ties were extremely common, so to measure inter-annotator agreement for the entire task, we would need to measure the similarity of three indeterminant ranked lists of sets (see Appendix for discussion and metrics definitions). However, as we were most interested in predictions at the lowest (i.e. most recommended) rank, we simply evaluated inter-annotator set similarity at rank 1 for each annotator. We find that the average Jaccard similarity of the set of rank 1 result(s) in each annotated engine-comparing comment was 0.86 for the two human annotators. Annotator 1’s mean Jaccard similarity with GPT-4 at rank 1 was 0.65, and Annotator 2’s mean Jaccard similarity with GPT-4 at rank 1 was 0.71. On Task 3, GPT4 was less accurate, treating Annotator 1 as ground truth. GPT-4 correctly identified three of the five queries but also returned three false negatives with a precision of 0.5 and recall of 0.6 (see Appendix for definitions). Task 3 annotations were only to help qualitatively identify evaluation strategies, and no quantitative metrics were derived or reported from task 3.