Science in the Age of AI: Knowledge, Data, and Trust

Alistair Nolan argues that AI is being used across every step of the scientific process and in every domain of science, functioning as a wonderful adjunct and amplifier of human intelligence. He is broadly optimistic about the long-term implications of AI for science, while acknowledging that there is considerable hyperbole surrounding it and that it will create institutional stresses.

He noted that AI is being used across every step in the scientific process and in every domain of science, describing it as a wonderful adjunct and amplifier of human intelligence , while also cautioning that there is a lot of hyperbole and that AI will create a series of institutional stresses .

Nolan argues that the primary challenge in fields such as materials science is no longer making discoveries but rather translating those discoveries from laboratory scale to industrial-scale production. He suggests that AI will help accelerate discovery, making the scaling and application of knowledge the new bottleneck.

He used the example of materials science and its contribution to addressing climate challenges such as battery technology and self-cooling buildings, noting that research his team conducted found that discovery is not the primary bottleneck - rather, it is translating laboratory findings to industrial-scale production of materials in thousands or millions of tonnes .

Nolan contends that AI will change who produces new knowledge and when, by freeing promising young researchers from time-consuming lower-level cognitive tasks. This will allow them to reach the frontier of their disciplines sooner and engage with genuinely novel problems at a younger age.

He cited the remarks of Fields Medal-winning mathematician Terence Tao, who argued that promising young mathematicians will now be able to reach the frontier sooner because less of their cognitive bandwidth will be spent on lower-level tasks such as memorising large bodies of literature and developing aptitude for long calculations and proofs .

Nolan argues that large technology companies are outspending public universities on AI R&D by significant multiples, drawing off talent and collaborating primarily with elite research institutions in the United States. This collaboration narrows the research agenda towards high-compute, large-data AI approaches that serve the companies' own assets, at the expense of the broader public interest.

He referenced a 2023 OECD publication showing that large tech companies tend to collaborate with elite US research institutions whose research profiles are much narrower than the broader university system, concentrating on types of AI that rely on high compute and large volumes of data - the very assets held by those companies . He suggested that more investment in smaller, less energy- and data-hungry models would be beneficial but may require some form of intervention .

Dziubek uses AlphaFold as a prime example of AI's transformative potential in science, noting that it earned its creators the Nobel Prize in Chemistry in 2024 by predicting three-dimensional protein structures. However, he stresses that the training data underpinning such systems consists of models reconstructed from raw experimental data, all of which carry inherent limitations and uncertainties.

He explained that AlphaFold uses deep learning algorithms trained on experimentally determined protein structures stored in the Protein Data Bank, which contains over a quarter of a million structures determined via X-ray diffraction, cryo-EM, or nuclear magnetic resonance . He noted that these structures are models reconstructed from raw data and have never been directly observed, meaning they carry all the problems that models have .

Dziubek argues that the ground truth used to validate AI models is not static but changes as new data sets emerge every day, meaning that what was considered accurate yesterday may not be so today. This creates a fundamental challenge for ensuring the reliability of AI-generated scientific outputs over time.

He stated that every day new data sets are produced and validation is checked against something different, so the ground truth of yesterday is not the ground truth of today . He described CODATA's current work on landscaping different measures for data quality across disciplines, requiring agreement on measures of uncertainty, accuracy, traceability, and provenance .

Dziubek draws on the aphorism attributed to statistician George Box — that all models are wrong but some are useful — to argue that scientific data used to train AI systems are themselves models with inherent flaws. He contends that the scientific community must actively seek to eliminate biased or low-quality models and demand full transparency about data quality from those deploying AI methods.

He cited the aphorism attributed to British statistician George Box that 'all models are wrong, but some are useful' , and explained that the scientific community tries to eliminate biased, low-quality, or contextually inappropriate models . He argued that if people using AI methods do not provide information on uncertainty, accuracy, traceability, and provenance, that is a big red flag .

Dziubek warns that AI will massively generate data, creating a bottleneck at which humanity will no longer be able to comprehend or validate what is being produced. This raises urgent questions about the quality and usefulness of the data, as well as whether all data should be open and fair.

He stated that AI will massively create data and that there is a risk of a bottleneck at which we will not be able to comprehend the data, describing the data deluge as a big problem at present and in the near future .

Dziubek argues that regardless of what AI predicts or generates — whether a new drug or a new material — the final test in experimental science is always the physical experiment or clinical study. This grounds AI-generated knowledge in empirical reality and provides the ultimate check on its validity.

He stated that in experimental science the final test is always the experiment, so if AI provides an answer about what drug or material is being designed, there is a clinical study or an experiment that then confirms or refutes it .

Dziubek argues that when data is not fully open, one must ask whether the dataset is representative and whether exclusions introduce bias. He contends that if restricted data causes bias in AI training sets, the consequences can be critical, particularly in high-stakes domains such as drug discovery.

He argued that the main threat of non-fully-open data is the risk of bias, and that if the excluded data causes the dataset to be biased, it can be really crucial, citing examples from language data, STEM data, and drug discovery data where input from different groups is essential .

Dziubek's colleague Dr. Vanessa McBride framed the panel discussion around three interconnected dimensions that the organisers identified as central to the topic of AI and science. These dimensions — knowledge generation, data foundations, and trust and integrity — provided the organising structure for the session.

Dr. McBride outlined the three dimensions as: AI and knowledge generation, scientific data and how foundational it is for AI, and the downstream issues of reliability, trust, and research integrity .

Thiga warns that while AI offers significant opportunities for leapfrogging in the Global South, it is simultaneously producing researchers who lack fundamental empirical skills. These scientists can generate publications with AI assistance but cannot conduct real experiments, critically read papers, collect data, or evaluate research outputs independently.

He described researchers in the Global South who cannot read a paper, conduct a literature review, collect data, analyse data, or critically evaluate the outputs of their research . He noted that this is compounded by academic and research leadership that is not conversant with AI and tends to demonise it, driving it into the shadows .

Thiga argues that AI is forcing a fundamental reconsideration of what universities are for, what research means, and what skills are needed in the age of AI. He suggests that institutions should prioritise better compute over better campuses and position themselves as ethical gatekeepers that teach good science and the scientific process.

He argued that institutions need to rethink what a university is for, what happens in the lecture hall, and what research means when AI can perform literature reviews and analysis . He suggested that institutions should focus on having better compute than better campuses and should teach ethics and good science, emphasising that there is a human at the end of every discovery .

Thiga contends that the true value of science is neither in the question nor in the answer, but in the method of inquiry itself — the process of observation, hypothesis formation, experimentation, data collection, and discovery. He warns that AI is at risk of stealing this beauty from science by short-circuiting the process.

He described the scientific method as encompassing inquiry, observation, hypothesis, experiment, data collection, and discovery, and argued that this is what AI is about to steal from science - the beauty of the inquiry . He framed this as the core of what scientists are losing in the age of AI .

Thiga argues that the scientific system has never depended on infallible scientists but rather on a system of checks and balances, and that the arrival of AI as a new player requires a rethinking of how those checks and balances function. This rethinking must cover ethics, data, compute, and the power imbalance between industry and academia.

He stated that science has never been about infallible scientists who cannot make mistakes but about checks and balances, and that the challenge now is to rethink how regulation works with AI as a new player, covering ethics, data, compute, and the power imbalance between industry and academia .

Thiga argues that only governments have the capacity to match the investment levels of large technology companies in AI, and that national sovereignty and pride may be the motivating force behind such investment. He points to India as a model of the kind of public investment needed to counterbalance industry dominance.

He stated that the responsibility falls at the floor of governments, as only governments can really match what some industry players are doing, and cited India as a case he finds fascinating for the kind of investment it is making to match industry, framing it as a matter of national pride and sovereignty .

Thiga acknowledges that AI offers the Global South opportunities to leapfrog in areas such as laboratory simulation and data analysis, but warns that the lack of data infrastructure, compute capacity, and native AI development capability means the region risks being left even further behind. The predominance of Western data in AI models compounds this disadvantage.

He noted that AI models are predominantly fed by Western data, that data systems in the Global South are nascent and not yet robust, and that the region lacks native infrastructure for compute, which must still be purchased . He described AI as presenting both leapfrogging opportunities and the danger of being left really far behind without the capacity to develop local models .

McBride highlights a significant gap in national AI strategies: while many countries are developing such strategies, almost none of them focus on the science sector itself. The strategies tend to address how AI can be applied in sectors like health and agriculture, but neglect to consider how AI is changing the science that will ultimately produce new technologies and applications.

She referenced an International Science Council report published earlier in the year, based on a synthesis of case studies across 26 countries, which found that while there are lots of national AI strategies under development, almost none of them have any focus whatsoever on the science sector itself . She noted that the strategies focus on health and agriculture applications but do not consider how AI is changing the science that will result in new technologies .

McBride framed the panel discussion around three interconnected dimensions that the organisers identified as central to the topic of AI and science. These dimensions — knowledge generation, data foundations, and trust and integrity — provided the organising structure for the session.

She outlined the three dimensions as: AI and knowledge generation, scientific data and how foundational it is for AI, and the downstream issues of reliability, trust, and research integrity .

Marivate draws on his direct experience as an area chair and senior area chair for major AI conferences to describe a crisis of integrity in scientific publishing within the AI research community itself. The volume of submissions has exploded, and there are serious questions about how many papers represent genuine scientific work versus AI-generated content with no real scientific goal.

He described how submission cycles in the Association for Computational Linguistics rolling review have grown to 13,000 papers in a single cycle, up from 2,000-3,000 a few years ago . He noted that reviewers are now checking whether references in papers are real, and that NeurIPS now attracts 20,000-30,000 participants . He questioned how many submissions represent genuine scientific experimentation versus purely AI-generated content .

Marivate argues that natural language processing and AI offer significant opportunities to advance knowledge about lower-resource languages, potentially uncovering new linguistic rules and knowledge that were not previously anticipated. However, he stresses that the way scientists wield AI as a tool — rather than allowing it to do their thinking for them — is critical to realising these benefits.

He noted that much of the work in building language models is still based around English or very high-resource languages, and that there is great opportunity in modelling lower-resource languages, potentially discovering new rules about different scripts that were not anticipated . He also reflected personally on the importance of sitting with one's own thoughts and using notebooks rather than immediately going to the AI prompt box, so that AI is used as a tool rather than a substitute for thinking .

Marivate argues that standard open data licences such as CC0 and CCBY allow large, well-resourced actors — particularly big tech companies — to extract value from community-generated data without reinvesting in those communities. He draws a parallel with the open-source software community's experience with copyleft licences and argues that equitable licensing models are needed to address this imbalance.

He noted that Creative Commons is currently undergoing a review because of the challenge of open washing, where people with the compute and engineering capacity accrue most of the benefits of openly licensed data without investing back into the communities that created it . He drew a parallel with the copyleft movement in open-source software .

Marivate describes two novel licensing frameworks — the ESETU licence developed by his startup Lilapa AI and the Noodle (No Letter or Bordeaux Open Data Licence) developed at the University of Pretoria — that attempt to address the inequity of standard open licences by discriminating by geography or identity. These licences aim to ensure that communities in the Global South retain some benefit from data they have created.

He described the ESETU licence, which allows those identifying as African to use data for commercial or non-commercial purposes freely, while requiring those who do not identify as African to use it only for non-commercial purposes or to negotiate benefit-sharing for commercial use . He also described the Noodle licence, which discriminates by whether the user comes from a developed or developing country and requires benefit-sharing from developed-country users .

Marivate argues that despite open science requirements from governments and institutions, data is already being because communities fear exploitation by well-resourced actors. He contends that the appropriate response is not to abandon openness but to fix the licensing framework so that it genuinely protects and benefits data-originating communities.

He noted that even with open science requirements, data is hiding, and people are figuring out ways to hide it even more . He cited the common practice of papers stating 'data available on request' but rarely delivering on that promise . He also referenced feedback from the Lacuna Fund, where communities reported that open data requirements led to big tech companies taking the data and releasing updates without recognition or benefit-sharing for the originating communities . He concluded that the solution is to fix the licensing framework, not abandon openness .

Marivate argues that the financial model of large AI companies, which spend vastly more than they earn on the promise of building an 'everything machine', is not a template that nations — particularly in the Global South — need to follow. Smaller, task-specific models can be highly effective and are a viable alternative for countries seeking AI capability without unsustainable expenditure.

He argued that nations do not need to build large AI models requiring enormous data and compute just to demonstrate capability, and that small language models or task-specific models - even a few megabytes in size - can compete effectively for specific scientific tasks . He described the rationale for the massive spending as being driven by the promise of an 'everything machine' that will solve all of humanity's problems, which he characterised as untrue .

Marivate warns that the AI bubble may burst — or may already be bursting quietly — and that humanity, especially the Global South, needs to build genuine scientific and technical capacity to survive beyond it. Dependence on large proprietary AI systems leaves communities vulnerable when those systems fail or become inaccessible.

He stated that the bubble is going to burst or is going to be somewhere that it has actually burst, and that humanity needs to survive past that . He argued that this means continuing to do basic high-impact research work and building genuine capacity, rather than depending on large proprietary systems .

Mercier argues that AI is not merely being applied to specific problems within disciplines but is fundamentally changing the very practice of science and the pursuit of knowledge. She illustrates this with the striking possibility that AI may come to understand the brain even when humans do not, raising profound questions about interpretability and whether scientific outcomes are acceptable without advancing human understanding.

She described an anticipation workshop on cognitive enhancement in which the conclusion was reached that by combining brain-computer interfaces with AI, AI may come to understand the brain even if humans do not . She noted that this raises the question of whether we are comfortable with AI achieving scientific outcomes - such as treating neurological diseases - without humans understanding how it does so .

Mercier highlights drug discovery as one of the areas where AI is having the greatest impact and where automation along the entire pipeline is most welcome. She notes that within a 25-year time frame, AI mining of clinical data and synthesis of chemical compounds could compress drug discovery timelines from years to days.

She cited an anticipation from one of her organisation's committees that within a 25-year time frame, drug discovery time could be cut from years to a matter of days due to AI mining of clinical data and synthesis of chemical compounds . She noted that this is an area where automation would be very welcome, in contrast to other aspects of science .

Mercier suggests a reframing of the relationship between data sovereignty and open science, arguing that the two need not be in opposition. She proposes that data sovereignty efforts may actually be part of the solution to making data open — not universally, but locally, to the networks and communities where it should be accessible.

She posed the question of whether data sovereignty is actually part of the solution to making data open to the local networks where it should be open, rather than being opposed to open science . She characterised this as her understanding of Marivate's answer about equitable licensing .

An audience member from the International Federation of Library Associations argues that the development of discipline-specific codes of practice and protocols is valuable because it demonstrates that the scientific community can regulate itself in a manner consistent with its own values. This is preferable to government regulation that may be poorly considered, either over-regulating or under-regulating science by accident.

The audience member noted the risk that national AI strategies might regulate science by accident - either sweeping it in with everything else or becoming overly cautious - and argued that developing codes of practice on a disciplinary basis would demonstrate the community's capacity for self-regulation . They asked what seems to work and what opportunities exist to accelerate the development of inclusive, updatable protocols and ethics .

Castle frames the data dimension of the discussion by highlighting that science is fundamentally a public good whose benefits should be universally accessible, yet the centrality of data to AI creates tensions between ensuring open access and protecting data sovereignty. He identifies this as a core challenge requiring careful balance.

He posed the question of what is at stake in thinking about science as a public good and a generator of knowledge that all people can benefit from, noting that at the core of AI is data, and asking how to protect against unfair or unwarranted access versus over-restrictive data sovereignty and protection of data .

Castle raises the provocative question of whether AI might generate scientific knowledge independently, or at least in a supervised or semi-supervised environment, inviting panellists to identify domains where this is already happening or likely to happen. This frames AI not merely as a tool for human scientists but as a potential autonomous contributor to knowledge production.

He asked the panel what areas they think AI might contribute to generating scientific knowledge, using the word 'independently' provocatively, while also allowing for the possibility of supervised or semi-supervised environments .

Castle opens the substantive discussion by directing the panel's attention to the long-term implications of AI's permeation of the research process for how scientific knowledge is produced. He frames this as a question requiring input from multiple perspectives represented on the panel.

He asked the panel to comment on what the long-term implications are for the production of scientific knowledge as AI continues to permeate various aspects of the research process .