A story of (non)compliance, bias, and conspiracies: How Google and Yandex represented Smart Voting during the 2021 parliamentary elections in Russia

On 3 September 2021, the Russian court forbade Google and Yandex to display search results for “Smart Voting,” the query referring to a tactical voting project by the jailed Russian opposition leader Alexei Navalny. To examine whether the two search engines complied with the court order, we collected top search outputs for the query from Google and Yandex. Our analysis demonstrates the lack of compliance from both engines; however, while Google continued prioritizing outputs related to the opposition’s web resources, Yandex removed links to them and, in some cases, promoted conspiratorial claims aligning with the Russian authorities’ anti-Western narrative.

search engines in Russia, on the eve of the Russian parliamentary elections in September 2021. • Despite being ordered by the Russian court not to display results for "Smart Voting," both search engines kept retrieving results for the query. However, Yandex did remove links to the opposition's web resources dealing with Smart Voting and, in some cases, prioritized conspiratorial claims aligning with the anti-Western narrative of the Russian authorities. This finding corroborates earlier research which suggested that Yandex is susceptible to governmental pressure and highlights the possibility of the corporation's algorithms enabling a less pervasive form of censorship in favor of the Kremlin. • Our observations stress the importance of scrutinizing how platforms implement government-driven content moderation, in particular in authoritarian contexts where declared anti-disinformation efforts can facilitate censorship. The presence of political bias in Google and Yandex outputs, together with conspiratorial information in the case of Yandex, also raises concerns about platform-wide filter bubbles and the current state of information freedom in Russia. This stresses the importance of mechanisms to monitor how platforms' algorithms distribute (politically) contentious content and how this process is affected by state regulation and other forms of political pressure.

Implications
The Smart Voting project was initiated by the currently jailed Russian opposition leader Alexei Navalny and his associates (Team Navalny) in 2018. Its aim is to undermine the political monopoly of the ruling United Russia party by preventing the dispersion of opposition votes across multiple candidates through voter coordination. The effectiveness of this strategy has been demonstrated, for instance, during the 2019 local elections in Saint Petersburg (Golosov & Turchenko, 2021). Via a dedicated website and the mobile app, Smart Voting shows voters which alternative candidate is most likely to defeat the candidate of the ruling party in their polling district. In the lead up to the parliamentary elections (September 17-19, 2021), the Russian authorities sought to limit the effect of Smart Voting: access to the app was restricted by the Russian telecommunications regulator Roskomnadzor on August 23, and the Smart Voting website was blocked on September 7. The Russian authorities also put pressure on Western and domestic online intermediaries to undermine the effectiveness of Smart Voting. Google and Apple complied with a government request to remove the Smart Voting app from their app stores (Lokot & Wijermars, 2021). At the same time, the Moscow arbitration court requested the two leading search engines in Russia, Yandex and Google, to remove Smart Voting from their search results. The request was presented as an interim measure in a case filed by the Russian company Woolintertrade, which had registered Smart Voting as its own trademark in July. Specifically, the court ruled to "Prohibit [Google/Yandex] the use of the 'Smart Voting' designation in search results of a search engine owned by the defendant [Google/Yandex] as one of the search keywords" (Elektronnoe pravosudie, 2021a, p. 1; 2021b, p. 1). How exactly the search engines should have implemented this order was not specified.
The growing pressure on both Google and Yandex ahead of the 2021 elections highlights the increasing importance the Russian government attributes to expanding its influence over online intermediaries, including search engines. While some years ago the Russian authorities aimed to increase their control over online information distribution by (unsuccessfully) trying to create a state-controlled and a state-owned search engine (Sanovich et al., 2018), by now their strategy has shifted towards putting pressure on existing platforms to make them comply with the regime's demands. In recent years, Russia's capacity to exert control over online information distribution has been strengthened through increasingly restrictive Internet regulation and enhanced state control over internet infrastructures (Ermoshina & Musiani, 2017;Sivetc, 2021;Stadnik, 2021).
Since Yandex is a Russian domestic corporation, it is generally thought to be more responsive to government pressure (Daucé & Loveluck, 2021) while Google is seen by many Russian NGOs as a protector of civil liberties (Bronnikova & Zaytseva, 2021) Compared with Google, Yandex was more intensively targeted by Russian regulatory mechanisms (Wijermars, 2021) and demonstrated more politically biased performance during periods of political contention (Kravets & Toepfl, 2021). However, the Russian authorities also put increasing pressure on Google as demonstrated by the growing number of requests for removing results from its search, particularly since the second half of 2020 ( Figure 1).

Figure 1. Number of Russian government removal requests for Google (web search). The graph is created using data from
Google's Transparency Report (Google, 2021).
Across all Google services, Russia is responsible for the highest number of removal requests (Google, 2021). While more than 90% of these requests are related to copyright issues, it is impossible to estimate what share of copyright requests might be politics-related. Furthermore, Google is increasingly complying with the Russian government's requests ( Figure 2). This trend seems to be specific to Russia: in the case of Turkey, for example, Google's compliance rate is around 40%, while for China it is close to 0% (Google, 2021). Given the limited information, it is difficult to ascertain whether the high compliance rate indicates a change in Google's treatment of Russian removal requests (i.e., becoming more likely to comply with them) or a qualitative change in these requests (e.g., a higher proportion of evidently illegal content which is requested to be removed).

Figure 2. Share of content removed by Google in response to the Russian government's requests (all services including web search).
The graph is created using data from Google's Transparency Report (Google, 2021). Note that the beginning of 2019 is the earliest period for which the data is available in the report.
Against this backdrop of growing governmental pressure, we looked at how Google and Yandex dealt with the above-mentioned court order and found that both search engines continued to display results for the "Smart Voting" query. However, Yandex did limit access to the opposition's web resources dealing with Smart Voting, which, presumably, was the main aim of the politically motivated court case. By contrast, Google did not remove Smart Voting from its search results and continued to link to Team Navalny resources. These differences in (non)compliance were amplified by political bias in the outputs, with Google outputs including more pro-Smart Voting content (and vice versa for Yandex), as well as Yandex tending to promote conspiratorial claims about Smart Voting coming from the Russian authorities (e.g., it being a project of the Pentagon aimed at harming the Russian people) and Google usually omitting such claims from its top results. There are several implications of our observations. First, the example of Yandex illustrates the importance of scrutinizing how compliance with government removal requests as well as governmentdriven content moderation is implemented. Particularly in the case of "informational autocracies" (Guriev & Treisman, 2020, p. 1) in restricted media systems, such as Russia, where the preservation of the regime is largely contingent on successful censorship, the selective removal of links from search results can be used to facilitate dissemination of misinformation and anti-Western conspiracy theories. It can also serve as a form of "masked censorship" (Makhortykh & Bastian, 2020, p. 14), namely a less intrusive way of filtering out undesired information. The potentially damaging effects extend far beyond the removal of the links themselves as information suppression can erode democratization processes (Stoycheff et al., 2018).
Under these circumstances, the decisions of online intermediaries concerning the implementation of removal requests play a key role in mediating the impact of censorship. Depending on these decisions, intermediaries might limit the visibility of outlets spreading extremist or false claims, but also facilitate state censorship by integrating it with "private censorship" (i.e., censorship effort by private companies) (see Beazer et al., 2021;Crabtree et al., 2015) or amplifying self-censorship practices (e.g., of Russian journalists in the aftermath of the annexation of Crimea) (Schimpfössl et al., 2020;Zeveleva, 2020). This raises questions about the feasibility of one-size-fits-all regulation of intermediaries' activities that would be applicable to their operations across democratic and non-democratic contexts, especially in the case of countries with long-standing censorship traditions (e.g., Russia and many other post-Communist states) (Ognyanova, 2019).
Second, our observations demonstrate the possibility of profound information inequalities forming between the users of Russia's two largest search engines. While the mere presence of such inequalities is not unexpected, especially in the context of the increasingly fragmented and polarized Eastern European digital ecosystems (Urman, 2019;Urman & Makhortykh, 2021a), the empirical evidence of their amplification by algorithmic mechanisms is concerning. Considering that these inequalities in the case of Smart Voting (and, potentially, other political matters in the region) are subjected to political bias, such amplification may encase Yandex and Google users into platform-wide filter bubbles (Pariser, 2011), in particular as the Russian search market is roughly split between Google and Yandex (Statscounter, 2021).
Third, while one might argue that, in the case of Smart Voting, Google search algorithms enabled a less censored selection of information, future cases may be less clear-cut. In particular, specific features of non-free media systems raise concerns about the universality of source prioritization principles. While both Google and Yandex tend to prioritize journalistic media (Zavadski & Toepfl, 2019), our observations demonstrate that their selection of specific outlets is subjected to political bias with the former prioritizing less state-dependent outlets and the latter giving priority to more Kremlin-oriented media.
It is less clear what Google will do if the number of independent media in Russia continues to decrease, considering the ongoing campaign against journalistic "foreign agents" (Roth, 2021, para 1). Will it shift towards prioritizing the state-sponsored media, following the general policies of the company which are also reflected in its algorithmic systems? Or will these policies be adapted for the case of Russia (and, potentially, other non-free media systems)? To answer these questions, it is important to keep monitoring both how platform algorithms distribute contentious political content and how such distribution is affected by regulation in both Western and non-Western contexts.
Together, our findings provide empirical evidence that may lay ground for a public debate on how and to what degree online intermediaries collaborate with authoritarian governments in enabling censorship. The importance of this debate is amplified by the ongoing transition from traditional violence-based autocracies to informational autocracies. Informational autocracies rely on the regime's ability to convince the public of its competency (Guriev & Treisman, 2020), which is a task increasingly fulfilled with the help of online intermediaries. Under these circumstances, it is essential to define what behavior is ethically desirable for the corporations working in non-democratic contexts. For instance, should Google collaborate with authoritarian states as it seems to increasingly do in Russia? Or should it non-comply as in the case of Smart Voting and risk fines and, potentially, withdraw from the regional market as in the case of China?
Answering these questions can be the first step for building up pressure from the public and policymakers to enforce the desirable behavior from the online information intermediaries. Similarly, it can help determine how intermediaries coming from authoritarian states and, potentially, serving as integral components of the informational autocracy shall be treated. Is there a point, for instance, at which Yandex search engine should be viewed as another information asset of the Kremlin, similar to Russia Today, and approached accordingly?

Findings
Finding 1: Google and Yandex kept displaying results in response to the "Smart Voting" query, despite being ordered to stop doing so.
Despite being ordered to stop displaying search results in response to the "Smart Voting" query, neither Google nor Yandex did so on the eve of the Russian elections. Both search engines kept providing the results throughout the whole period of data collection (September 10-20). This non-compliance can be primarily attributed to the technical implausibility of the initial request. Unlike the requests for Google and Apple to remove the Smart Voting app from their App Stores (Lokot & Wijermars, 2021), which occurred around the same time, the technical realization of non-displaying the results for a particular query is a non-trivial task. While modification of search outputs is not impossible, it usually involves filtering out individual websites from the outputs in general or for specific, but more narrowly defined, queries (e.g., as in the case of the implementation of the right to be forgotten, Mangini et al., 2020).
The imprecision of the request was also noted by the Yandex representatives, who remarked that "it is unclear what exactly we are requested to do [by the court] and how we can do it" (Interfax, 2021, para 2). Google did not provide any commentary on the decision, but it can be presumed that it also found the court decision difficult to implement. The fact that Yandex stopped prioritizing the links to the Team Navalny resources can be viewed as a form of compliance (albeit more with the legal requirement to stop displaying links to blacklisted websites rather than the interim measure from the trademark case).

Finding 2: Google and Yandex prioritized different types of sources.
Contrary to earlier studies which did not observe substantive differences in the selection of sources by Yandex and Google (Zavadski & Toepfl, 2017), we discovered substantial discrepancies in the content prioritized by the two engines, with the exception of journalistic media which dominated the outputs in both cases (Figure 3). Google consistently returned links to the Team Navalny resources related to Smart Voting, including the main Smart Voting website that the Russian government had blocked. Yandex did not return any outputs affiliated with the Team Navalny with the exception of September 13, when a link to the Smart Voting website appeared in the results once. We suggest that this exceptional behavior can be attributed to the output randomization used by search engines to maximize user engagement by testing different ways of ranking search results. Sometimes referred to as "Google Dance" (Battelle, 2005), this phenomenon remains largely under-studied (for some exceptions see Haim et al., 2016;, but it can be one of the factors influencing changes in the visibility of individual resources. We also observed differences in the prevalence of blogs and social media in search outputs. On Yandex, between 20% and 40% of top links were to blogs, while on Google only a few links to blogs appeared during the selected period. For social media the situation was reversed: Google returned more links to the platforms, such as Facebook and YouTube, than Yandex. There also were differences in the affiliation of blogs and social media accounts. Whereas the blogs, in the case of Yandex, usually belonged to anonymous individuals who often promoted pro-Kremlin narratives, the social media pages prioritized by Google usually belonged to Team Navalny. The share of links to reference websites (e.g., Wikipedia) was similar across the two engines. Finding 3: Content prioritized by Google and Yandex was politically biased.

Figure 3. Shares of links to content of different types returned by Google (above) and Yandex (below), by date. The numbers on the bar segments denote the exact share 3 of links of a specific type for all rounds of data collection on
We found major differences in the share of content with different stances towards Smart Voting ( Figure  4). Both search engines returned a similar share of links to neutral content (hovering around 50% of all links), with Google having a higher share of neutral content at the beginning of the data collection period than Yandex. However, the non-neutral content distribution was very different: in the case of Google, almost all non-neutral content had a positive stance towards Smart Voting, while on Yandex the share of pro-Smart Voting content never went above 30% and was consistently lower than the share of anti-Smart Voting content. Thus, we observe a political bias in the outputs of the two search engines. Among positive and negative assessments of Smart Voting a broad variety of perspectives was found. In the case of anti-Smart Voting content prioritized by Yandex, we observed claims that Smart Voting is a form of manipulation aiming to "deprive the citizens of their right to vote" (Babin, 2020, para 1) or a security threat to personal data (REN TV, 2021a), as well as arguments presenting Smart Voting as ineffective or morally questionable (e.g., because it often recommended voting for the Communist Party) (Novye Izvestiia, 2021). In the case of Google, a positive stance was often expressed in the form of independent media either hosting op-eds that endorsed Smart Voting or linking to the Smart Voting candidate lists.

Figure 5. Shares of content with conspiratorial claims about Smart Voting returned by Google (above) and Yandex (below), by date.
The numbers on the bar segments denote the exact share of links of a specific type for all rounds of data collection on a particular day. For example, out of all Google results collected on September 11, 75% linked to web pages without mentions of conspiracy theories, 16% linked to pages with conspiracy-promoting content, and 9% linked to pages that mentioned conspiracies without promoting or debunking them.
In some cases, search outputs also included conspiratorial claims related to Smart Voting ( Figure 5) such as that it is aiming to harm Russian voters (Lenta, 2021) or is a U.S. Pentagon project designed to interfere in the Russian elections (REN TV, 2021b). Often, these claims originated from Russian officials such as the spokesperson of the Ministry of Foreign Affairs, Maria Zakharova, or the spokesperson of the Russian President, Dmitrii Peskov, that also resulted in them being reiterated by authoritative sources (e.g., established journalistic media). While both Google and Yandex included such claims in their top results, Google tended to prioritize content without mentions of conspiratorial information (e.g., stories about Smart Voting coming from BBC or RBC). By contrast, Yandex gave more visibility to items promoting conspiratorial claims which were usually coming from pro-Kremlin outlets (e.g., TASS or REN TV).

Data collection
We conducted an agent-based audit of the first page of search results of Yandex and Google for the query "умное голосование" (Smart Voting) (for further details on the chosen auditing approach, which seeks to simulate human browsing behavior, see Appendix, Section A). This specific query was chosen because it is the term to which the court order applies. While users may have also used other search terms, examining the content retrieved for such related (e.g., "SmartVote app") or derived queries (e.g., "Smart Voting Navalny") is beyond the scope of the court order and therefore this study.
Data collection started one week before the start of the voting period (September 10) and ended on the last voting day (September 20). To increase the robustness of our results, we aimed to have four rounds of the data collection each day at the same time; however, due to technical issues (e.g., a change in the underlying HTML code of Google's search page) on September 14 we had only one round of data collection, on September 19 two, and on September 16 three rounds instead of four. Notwithstanding this and a few other limitations (see Appendix, Section B), we suggest that our results are robust enough to infer how information about Smart Voting has been distributed by the two engines.

Data analysis
We extracted all hyperlinks gathered from the organic results on the first page of Yandex and Google search results. Altogether, we collected 731 hyperlinks. The odd number is due to search engines sometimes providing fewer than ten organic results on their first page because of the presence of other elements (e.g., "Knowledge Panels" or "People also ask"). Among the 731 results, there were 119 unique hyperlinks. We manually coded each hyperlinked page into the following categories: • Whether the page contains information related to Smart Voting: a binary variable (yes/no).
Then, for pages containing information related to Smart Voting we used two additional coding categories: • Whether the page content is politically biased: 1) arguing in favor of Smart Voting, 2) arguing against Smart Voting, 3) neutral regarding Smart Voting. • Whether the page contains statements about Smart Voting that can be classified as conspiratorial.
For example, statements that present Smart Voting as part of secret agenda pursued by domestic (e.g., the Kremlin) or foreign (e.g., the US) actors to interfere in the elections and, potentially, harm the Russian people, were coded as conspiratorial: 1) the page promotes such statements, 2) the page debunks them, 3) the page mentions such statements without adopting clear stance towards them, or 4) the page does not mention any conspiratorial information.
The coding was performed independently by two coders, both fluent in Russian and knowledgeable of the Russian political context (on intercoder reliability, see Appendix, Section C). Finally, we matched the links collected for each of the search engines with their classifications and computed the shares of different types of content distributed by each search engine.

Competing interests
All of the authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics
Because our research did not involve data collection from human users or any interaction with human users, it was exempt from the ethical review under the guidelines of the Ethics Committee of the Faculty of Business, Economics and Informatics, University of Zurich. Further, since we ran only a few agents that did not perform actions such as clicking on specific items, we believe that our study did not interfere with the search algorithms in any potentially harmful way or affect the engagement statistics for individual results or put a heavy load on the search engines that could potentially affect their functionality.

B. Limitations
The conducted research has several limitations. The use of remote vantage points (e.g., VPNs) has been criticized for being not reliable (e.g., not being located in the advertised countries) (see Weinberg et al., 2018). While we verified that the vantage point was, indeed, located in Saint Petersburg, future studies can benefit from alternative approaches such as recruiting crowdworkers from the respective region to run either the search queries directly or the scripts for powering the agents. An additional benefit of using crowdworkers can be the possibility to examine actual search behavior (e.g., by asking crowdworkers to use not a pre-fixed set of queries but come up with their own search suggestions) and its interactions with search censorship and not just the behavior of engines in response to a query. Another limitation concerns the implementation of the auditing method used to conduct the study. The current study relies on a single search query and uses agents deployed via a single browser. While we assume that Google and Yandex did not censor other queries related to Smart Voting because of not being explicitly requested to do so, it would be worthwhile to empirically check the validity of our assumption. Similarly, further research can examine whether there are cross-browser differences in retrieved results, which is a phenomenon observed by some earlier studies (e.g., Urman et al., 2021).

С. Intercoder reliability
To evaluate the intercoder reliability, we compared the results produced by the two coders in the course of the original coding. There were no disagreements between the coders with regard to whether a page was related to Smart Voting as well as the page types. For the last two categories -political bias and conspiratorial information -the coders agreed in 80% (24 disagreements out of 119 hyperlinks) and 78% (26 disagreements out of 119 hyperlinks) of cases respectively. These disagreements were resolved through consensus coding. Finally, we matched the links collected for each of the search engines with their classifications and computed the shares of different types of content distributed by each search engine.