Post-Post-API Age: Studying Digital Platforms in Scant Data Access Times (2025)

2.2.1. Pre-API and Voluntary-API Eras

With the emergence of major social media platforms like Facebook (founded in 2004), YouTube (founded in 2005), and Twitter (founded in 2006), both companies and researchers recognized the immense value of digital data as a political and economic resource(Lazer etal., 2009).However, during this early period, which we term the “pre-API age,” programmatic approaches to accessing platform data remained limited.

Starting around 2010, researchers experienced a “golden age” of data access, as many social media platforms voluntarily developed APIs (the “voluntary-API age” in Figure1).One of the most influential examples was Twitter’s v1.1 and 2.0 APIs, which were free and open to all users and allowed accessing a wide range of data types with minimal restrictions. The Twitter 2.0 API also had an “academic track” that provided increased access for verified academic researchers.Over the decade, these APIs enabled countless research projects across various academic disciplines(Murtfeldt etal., 2024) and have been used for creating public-service tools and educational resources.Other researcher access programs from the same era include the Social Science One program(King and Persily, 2020), access through CrowdTangle, and the Meta U.S. 2020 elections Project(Wagner, 2023).These programs offered researchers diverse data types and varying levels of access, providing unique insights into different aspects of online platform behavior and user activity.

However, all data access programs have their limitations.Although APIs offer powerful and flexible access, platforms maintain complete control over the data access process, available data types, and API functionality(Bucher, 2013).This means that platforms can modify API specifications or end the API at their discretion, potentially disrupting ongoing research projects and tools.CrowdTangle also exemplified these constraints—while it provided relatively open access to public posts from Facebook and Instagram, it limited access to content from larger public pages and groups with over twenty-five thousand followers.The Social Science One initiative fell short of expectations, providing researchers with data of limited utility compared to what was initially promised(Alba, 2019).The project was further compromised when it was discovered that half of the dataset was missing due to production errors(Alba, 2021).At the same time, Facebook maintained control over both CrowdTangle and Social Science One, with the ability to deny data access to researchers whose work might not align with company interests.Similarly, the Meta U.S. 2020 election projects remained dependent on Facebook’s voluntary cooperation to provide data access.Notably, no subsequent projects with comparable arrangements were implemented or proposed, highlighting the unsustainable nature of this approach.

2.2.2. The Post-API Age

The aforementioned limitations collectively undermine the stability and reliability of data access approaches.A pivotal event in the history of platform data access was the Cambridge Analytica scandal in 2018.In response to this scandal, Facebook significantly restricted access to its Graph API, dealing a severe blow to the social media research community.In the influential essay “Computational Research in the Post-API Age”(Freelon, 2018), Freelon analyzed the implications of Facebook’s decision.The restrictions on the Graph API effectively eliminated all Terms of Service-compliant methods for systematic collection and analysis of Facebook data at the time.This shift led Freelon to characterize the resulting era as the “Post-API” age for Facebook research.

Another driving factor that led to more restricted data access is the rise of large language models (LLMs).The development of LLMs has been driven by the so-called “scaling law,” which states that the performance of LLMs is a function of the model’s size and the amount of training data(Hoffmann etal., 2022).This pattern makes user-generated data a valuable resource for training LLMs.As a result, online platforms have become increasingly cautious about granting data access.

Since 2023, the data access landscape has undergone a series of increasingly restrictive changes.Twitter initiated this trend by discontinuing its free API access.While a paid API remains available, its pricing structure has become prohibitively expensive for most researchers, with costs ranging from $100 per month for 10,000 tweets to $5,000 per month for one million tweets.Following Twitter’s lead, Reddit implemented similar restrictions on its free API access.Subsequently, in 2024, Meta announced the discontinuation of its CrowdTangle platform.These significant changes collectively mark a definitive shift in the “post-API” era for social media research.

In response to these challenges, researchers have intensified their exploration of alternative data collection methods beyond platform-provided APIs.Freelon’s essay remains particularly salient, emphasizing that effective social media platform research necessitates diverse data collection strategies, encompassing both collaborative and adversarial approaches.Many researchers have adopted unsanctioned web scraping as an alternative method(Bruns, 2021).However, this approach raises significant concerns as it introduces complex ethical and legal challenges that researchers must carefully navigate(Brown etal., 2024).As an alternative to scraping, researchers have developed more transparent data collection methods that incorporate explicit participant consent.For instance, Breuer etal. (2023) demonstrate how researchers can gather Facebook data through voluntary data donations from study participants.Furthermore, the emergence of open-source tools for facilitating and analyzing user-donated data has established data donation as an increasingly viable research methodology in the post-API era(Araujo etal., 2022).

4.2.1. Barriers to Applying

One of the barriers mentioned by researchers is the application process. Aligning with the survey results, some interviewees said they chose not to apply for API access for certain platforms because they found the application problematic.This sentiment was most glaringly obvious when they spoke about X (Twitter).Some researchers felt discouraged to apply or pay for the X API as they perceived the platform’s business practices under Elon Musk as going against democratic values.Bob, an information scientist in the U.S., decided to move away from research on X because he finds it ethically wrong to support the platform’s business in any way.“It’s now hostile to my values, to research, to knowledge …… it’s no longer a great research site for me to study the things I want to study,” said Bob.

When researchers applied for APIs, they described the application as cumbersome, especially as each platform had a different API application process, and the experiences of the researchers varied.Researchers lamented that the processes often did not reflect the realities of how research is conducted, were unnecessarily laborious, and required excessive time and resources.

For example, researchers are expected to apply for API access for individual research projects rather than for long-term access under the current data access regime.Researchers felt that this application was not designed to facilitate flexible uses of APIs.Max, a U.S.-based academic researcher who manages a large-scale research lab, explained that this requirement makes it hard for researchers to use the APIs simultaneously to conduct multiple projects.Discussing Meta and TikTok, Max said:

These challenges highlight a mismatch between current data access procedures and the pragmatic practice of conducting research using social media data: data access limited to individual projects makes it difficult for researchers to deftly study rapid and recent trends in the social media ecosystem.

Other barriers to access involved the arduous application process. For example, researchers who applied for the Meta Content Library API in the early phase said they were required to have ethical approval from Institutional Review Boards (IRB) for their projects. Although this requirement was later removed, the researchers indicated that it discouraged many scholars from applying because (1) such a practice hinders researcher independence and (2) data collection, preliminary data analysis, and the process of designing research often go hand in hand. Brian, a U.S.-based information scientist, welcomed Meta’s decision to drop the IRB requirement, but added:“[T]he fact that they even required it in the first place was a very limiting thing.”Furthermore, for some social scientists who are not based in the U.S., the IRB requirement was simply difficult to meet because obtaining ethical approval is not a common practice in their countries.Elaborating on this point, Fabio, a European political communication researcher, explained:“In my university, like in many, many other universities in Europe, this kind of ethical committee [is] …… not really common in social science.”

Other researchers mentioned that they struggled with a long list of questions and what they considered unrealistic criteria they were required to meet.Bastien, a researcher at a European civil-society organization, who applied for the LinkedIn API, recalled being asked around 40 questions at the time of his application.The platform “sets a very high bar” with its requirements, including what he perceived as overly strict privacy protection requirements for public data, said Bastien.While he agreed that platforms needed to ask some of these questions to ensure ethical data usage, Bastien felt that “there are almost no organizations that would actually be able to meet all these criteria.”He added, “the requirements around safety, security, privacy are absolutely disproportionate to the sensitivity of the [public] data.”

Crafting answers to these lengthy lists of questions can be highly time-consuming, which in turn slows down research.Andy, a E.U.-based media studies scholar, described the application forms for Meta’s tools as “pretty clear and straightforward” but felt that the applications were “unnecessarily onerous,” which “slows down your research project.”Exacerbating the issue, aligning with the survey results, interviewees reported that it took them weeks, or sometimes months, before they heard back from platforms for their decisions.While researchers who obtained TikTok API access reported what they considered a relatively short time frame before being able to use the API, that was not the case with Meta: they had to sign a legal agreement with the platform, which they described as a cumbersome process requiring legal assistance and considerable time resources. As Max put it:

In sum, although API application processes and experiences vary by platforms and by researchers, our interviews highlight that the applications generally seem to have become much more time-consuming and complex under the current regime, requiring researchers to invest more resources.This change made many researchers feel that platforms fell short of adequately supporting research endeavors.For researchers not based in the U.S., some requirements, such as IRB, made the application process even more cumbersome, as these requirements seem not to reflect research practices at non-U.S. institutions.As Ista, a E.U.-based computer scientist, put it: “They don’t understand the needs of the research community well enough, or what they need to do to meet them.”

4.2.2. Barriers to Access

Unfortunately, many interviewees were denied access to research APIs after going through these application processes.They either found the reasons for access denial provided by platforms unreasonable or felt that they were not given adequate explanations, believing that their research initiatives fell under what the European Commission calls research on systemic risks affecting E.U. citizens.In worse cases, researchers did not receive any communication from the platforms on their application statuses.

The majority of the rejected cases in our interviews involved trying to access X data.Erkki, a digital social scientist and a Ph.D. student based in the E.U., shared that his application for X’s academic research access was declined.Erkki has extensively studied climate discourses on X since before Elon Musk acquired the platform and applied for the X API to conduct research on how parliamentarians and municipal electoral candidates communicate about climate change.Although he thought there would be no reason for his application to be declined because he had access to the previous versions of the Twitter API and used them for similar research projects, the platform denied him access on the grounds that his application was “incomplete or lacks sufficient detail to show” that he met the criteria under the DSA, according to the email from X Developer Team shared by Erkki.Yet, “I don’t really see any reason why they would reject it this time,” said Erkki, adding, “obviously it felt like an injustice has been done.”

Ista (AR/E.U.) also applied for X’s academic research access with an eye to studying hate speech and counter-speech on the platform.After exchanging some emails with the platform to provide requested additional information, her application was denied without a “very consistent reason”:

Some researchers never heard back from the platforms, which they interpreted as effectivly a rejection.The lack of transparency and communication from the platforms left researchers in limbo, without knowing whether they would be able to carry out their research projects.Kay, a communication scholar at a public university in the U.S., applied for the Reddit API when it was first announced in 2023.But she “never heard anything back,” said Kay, adding that such a case is “not unique to me.”Corroborating Kay, Thirteen, a computer scientist in the U.S. studying social movements, said he only found out that his Reddit API application was rejected when he attended a workshop where a Reddit employee indicated that the decisions had been made.The information was only “disclosed informally at the [workshop], not in a transparent, public forum,” said Thirteen, adding that he was unsure why his application was rejected.

Researchers who are not affiliated with universities face even more limited data access because some platforms prioritize academic researchers.Different treatments of academic and non-academic researchers are not in alignment with Article 40 of DSA, which stipulates that researchers, including those at non-academic institutions, who are approved by national public authorities, must be granted access to platform data(deCarvalho, 2024).

TikTok is one of the platforms that, at the time this research was conducted, excluded non-academic researchers from API access.Bastien (NR/E.U.) applied for the TikTok API while appealing to the regulators of the country he was based in to obtain support for pushing TikTok to approve his application.However, the application was declined on the grounds that he was not affiliated with a university:“They told us, ‘right now, it’s not available to non-academics, but we are planning on releasing it wider soon and we’ll let you know.’ They never let us know,” said Bastien.He sent a follow-up email to TikTok, but received no response.Similarly, Devin, a computational social scientist at a publicly-funded European research institution, was denied API access on the same grounds.Devin thinks the reason for rejection is not justifiable because the Digital Service coordinators he had a meeting with made it clear that publicly-funded research institutions should be able to access data through platform APIs.“We’re not a university, it is true, but that’s not a reason that we do not fall under the DSA,” said Devin.This treatment of researchers not affiliated with universities is, in Bastien’s words, a “mismatch between what they offer and what the law says they have to offer [under] the DSA.”

Another interviewee, Green Wave, a researcher at a civil society organization in Latin America providing support for marginalized populations targeted by harmful social media activities, also voiced frustration about TikTok’s exclusion of non-academic researchers.To save time and resources, she decided not to apply for the TikTok API, given that it was designed mainly for academic researchers in the U.S. and E.U.However, not having access to this infrastructure affected her work, and she felt that platforms giving access to a limited pool of researchers went against the spirit of Article 40 DSA:

Stepping back, our interviews highlight that the decision-making processes regarding API applications are far from transparent.In many rejected cases from X, the platform’s justification was that these research projects fell outside the scope of the DSA, without elaborating on why or how.Some researchers never received responses from the platforms, leaving them to speculate about the reasons for denial.For researchers not affiliated with a university, access is even more limited because the door is closed from the beginning.Although some researchers said they would try to apply again or find alternative ways to collect data, not having official API access can limit the amount and the kinds of data scholars can use, narrowing the scope of their research significantly (we will come back to this point later).

4.2.3. Barriers to Usability

Despite the challenges mentioned above, there are cases where researchers were granted API access.When researchers gain access to APIs, what are their experiences? Are they satisfied with how the APIs function and the data they receive? Unfortunately, the answer is currently no—most of our interviewees reported encountering various obstacles while using the APIs.We observed three primary issues that made it difficult for scholars to conduct research using APIs: (1) difficulty collecting data in the first place, (2) restrictive data caps, and (3) poor data quality.

4.3.1. Web Scraping

Many participants noted that, despite changing data access procedures, there were still unofficial methods for collecting social media data. Peter, a European academic researcher, said, “if there are APIs, it’s nice, else I will try to find my own way of getting data.”Similarly, Sarah, a European digital studies researcher, noted that data collection is possible without APIs: “Python scripts where you can get Instagram data with. There’s also still the possibility of in-browser scraping, which is done for Twitter.”However, these alternative collection approaches also had limitations, such as “temporal gaps,” meaning “you cannot go into historical data well,” added Sarah.

One common alternative approach mentioned was data scraping.Some platforms, such as Alphabet (which owns Google and YouTube), allow permissive scraping, as Bastien (NR/E.U.) was able to get permission for: “What they grant you is the right to scrape YouTube basically from a given IP address.”However, in most cases, the legality and ethical ambiguity of scraping have raised concerns.For example, Kate, a European computer science researcher, said, “There are a lot of scraping tools even now with which you could scrape Twitter, but I’m not sure if it’s ethically okay to do it for research.”In other words, even if researchers could scrape data (and in many cases, they were able to), researchers wanted to ensure that they did so legally and ethically.Kay (AR/U.S.) said, “I will go to my grave saying scraping is not a crime, but it does go against the terms,” highlighting this difficult balance between doing important research on public data and adhering to terms of service obligations.She said she still collected data, but set herself some boundaries: “I do still scrape web forums that are open, that don’t require logins and websites.”Researchers like Devin (NR/E.U.) also noted that they were more comfortable scraping data about aggregate information (such as engagement), rather than individual-user data: “We didn’t look at user-level data. But instance-level data. So we scrape data. And that’s also data that I think is safe to consider [a] public right.”

Others said they would avoid using scraping as a data access strategy altogether because it is not the most reliable means of collecting data.For instance, Erkki (AR/E.U.) said he “really wouldn’t trust” any longitudinal or historical research relying on scraping because it is hard to obtain comprehensive data.Similarly, Fabio (AR/E.U.) said he is wary that the code used to scrape data could suddenly stop working because “when you are scraping, you are basically getting data in a way that is not supported by the platform.”Fabio added that the issue is not only the lack of official platform support for scraping but also the barriers they could put up to stop scrapers (e.g., rate-limiting traffic, detecting and blocking bots, or even taking legal action).Scraping means it is “not only not supported [by the platform] but sometimes actively fight [the platform],” said Fabio.

4.3.2. Other Approaches

There were many other alternatives that researchers considered other than scraping.Some used third-party applications and tools, such as Zeeschuimer, a browser extension; Apify, a web-scraping and automation cloud platform; and Brandwatch, a social listening tool. Others used existing archives of past data.Kate (AR/E.U.), for example, noted that she was using data she had collected previously: “I’m working with archival data that I have collected because it’s no longer feasible for researchers to get data [through APIs].”While these approaches have their own individual advantages and disadvantages, they collectively rely on some resources that are not democratically available to all researchers.Barriers include money (to buy access or to pay participants), coding proficiency (to build or use a scraper), or the difficulty of obtaining comprehensive historical or longitudinal data.

Because alternative data collections are imperfect, but official access is not available, many expressed concern that important research was not being conducted.Discussing the disadvantages and challenges of using alternative data access methods, Fabio (AR/E.U.) lamented, “You need to have a reliable way to assess this data …… this is really unfortunate because we don’t really have scraping [or other reliable and equitably accessible tools] and we don’t really have the data from official channels.”Bastien also expressed frustration that third-party vendors “don’t necessarily have the data we need and the format we need” and such tools usually provide data that is collected in a “less systematic way than what we would have with the APIs.”

4.5.1. Inflection Point to do Better Research

Many interviewees shared the view that social media research is at a critical juncture with the loss of free APIs.While the current conjuncture is often described as the “API Apocalypse” (Sarah, AR/E.U.), researchers also discussed ways to conduct better research within the current limitations, critically reflecting upon how the older data access regime has shaped their research fields.Some researchers argued that social media research has tended to concentrate on platforms with easier access to data, which they said has led to overlooking other important avenues for communication.They agreed that Twitter (X) was a popular platform for research, partly because data was readily available. Peter (AR/E.U.)’s following quote illustrates this point well:

Max (AR/U.S.) also saw the potential of producing more research on other relevant platforms as “the bright side” of the ongoing discourses around data access under the DSA.“Twitter was the platform where this data was most available …… but as people are moving to new platforms, especially because these platforms are open, then we hope we can do that again and use the kind of data on other platforms,” added Max.

Others argued that researchers should consider the possibility of small data research instead of focusing only on large datasets.For instance, Sarah (AR/E.U.) said that, with small data, researchers can make sure the data is reliable by manually checking if there are any discrepancies in data collected through APIs or scrapers.Although the loss of free APIs has limited the scope of research, it “also points out what you still can do even with small data …… you’re much more inclined to zoom into certain niches,” added Sarah. As these interviewees’ quotes indicate, some researchers are exploring new opportunities for social media research despite the lack of adequate data access under the DSA.

4.5.2. Collective Action and Policy

At the same time, however, researchers emphasized that platforms are responsible for ensuring equitable and adequate data access and that the research community needs to collectively pressure the platforms to step up their efforts to support research.They argued that platforms currently place strict limitations on who can access what kinds of data, as we have described above, and that they have not sufficiently considered the needs of the research community.

Brian (AR/U.S.) said that the current data access discourses are primarily driven by “the language of privacy, data minimization, protecting human subjects”, the rhetoric that platforms could weaponize to limit data access as much as they can.While acknowledging that “all of that is obviously an important consideration,” Brian indicated that a healthy balance between these ethical issues and ensuring data access to conduct important research is critical.“The bureaucracy is not serving the science [and] we’re not able to actually answer meaningful questions” under the current framework, added Brian.Researchers also voiced concerns that the current data access framework may compromise the independence of research from platform influence, calling for a community-wide discussion on the issue.

The current API application process that requires researchers to provide the details of their projects and the lack of transparency in platforms’ decision-making processes were of particular concern among researchers, as platforms could easily deny API access to research that could put pressure on the platforms to change their practices.Researchers should call on platforms to “make independent research possible within the regulatory framework,” said Sarah (AR/E.U.).To that end, researchers said it is paramount to build a joint effort to lobby for more ethical and sustainable data access.Fabio (AR/E.U.) argued that “the creation of a community of scholars and projects …… is super important from this perspective, because I’m always dealing with platforms in a point-to-point relationship.”

4.5.3. Types of API Data Desired

As we have described in Section4.3, researchers are eager to explore alternative data collection methods to overcome the current limitations of APIs, but there are significant limitations of such approaches.For many social media scholars, reliable and sustainable API data access remains a desired means to conduct novel and creative research; researchers spoke about what types of data they wish they could access through APIs to answer their research questions.In this subsection, we list the desired API data mentioned by our interviewees in the hope that it will serve as a step toward encouraging platform-researcher dialogue on improving data access.

One type of data often mentioned by researchers is information that is already publicly accessible or data on public figures (e.g., politicians, celebrities, and news organizations).Researchers argued that platforms should be able to make public data readily available for all researchers, regardless of their affiliations, given that there is less concern about privacy.Bastien (NR/E.U.) shared that he and his colleagues have mainly requested publicly available data from different platforms, but these attempts have been unsuccessful so far, adding that the bar is higher for non-academic researchers like him.“Ideally, we would want to have access to an API that was broadly like the (older) Twitter API,” but if that’s not possible, “what we would have liked was the possibility to input one account and get a list of all of their publicly accessible content for the last N years, along with a number of interactions,” added Bastien.Emily (AR/E.U.), who studies TikTok posts of high-profile comedians, said platforms should loosen limitations on data related to public figures because “their data is in public domain” and it should be “OK for us to have it or even allow researchers to share the data with other scholars.” An important effort to develop protocol for determining what counts as publicly accessible content has been advanced by the Knight-Georgetown Institute’s Gold Standard Working Group.⁵⁵5https://kgi.georgetown.edu/expert-working-groups/gold-standard-expert-working-group/gold-standard-faq

Another request we heard frequently is access to larger-scale data.Many researchers emphasized that large-scale data access is crucial for understanding trends, user behaviors, and platform activities (e.g., content moderation), and information flows on a platform-wide scale.“Without APIs, I’m never going to be able to create very big data sets,” said Wellstone, a researcher at a for-profit company.This limitation forces researchers to work with smaller samples that may not be representative, making it hard to draw reliable and generalizable conclusions regarding platforms.For example, Sarah (AR/E.U.) said she has focused on small data research on TikTok and such an approach can contribute to important research findings, but scalability is hard to achieve without the API:

Like Sarah, many other researchers discussed the types of API and data they would want from TikTok, in part because the platform has grown popular in many countries. This data included visual captions embedded in the videos (Emily, AR/E.U.), commenter identities that allow the creation of interaction networks based on a given content (Max, AR/U.S.), and all posts from a given account (Thirteen, AR/U.S.).

Overall, these findings highlight a dilemma: while researchers hope to rely less on platforms for data to secure independence, they also recognize limitations of alternative tools and the potential usefulness of official APIs.