Sakana AI's 'AI Researcher' automates scientific research

Tokyo-based artificial intelligence startup Sakana AI has today unveiled what it claims is the world's first generative AI model designed to perform scientific research on its own. The model, called AI Scientist, is the result of a collaboration between researchers from the University of Oxford and the University of British Columbia and is said to have the potential to automate many of the processes involved in scientific discovery.

According to the creators, AI Scientist is based on large language models capable of mimicking the scientific discovery process, starting with idea development, followed by conducting experiments, and ending with writing full scientific papers describing their discoveries.

"We propose and run a fully AI-powered system for automated scientific discovery applied to machine learning research," Sakana AI's team wrote in an academic paper describing the new model.

The study describes how the AI Scientist is designed to specifically conduct research in machine learning and is able to propose ideas such as new techniques for transformer-based language models, diffusion models, and analysis of learning dynamics.

Sakana AI claims that AI Scientist can produce a complete research paper at a cost of only $15, based on the computing resources it uses. Furthermore, its articles "exceed the acceptance threshold" of a well-known machine learning conference, as assessed by an automated review process.

"From idea development, coding, performing experiments and summarizing results, to writing full papers and conducting peer-review, AI Scientist opens a new era of AI-powered scientific research and accelerated discoveries," the researchers wrote.

In a blog post, Sakana AI's team said that AI Scientist is able to apply a broad research direction and starting code base, such as the open source code base of older research on GitHub, and begin its own discovery process. They also claimed that the AI model can follow the procedure of AI researchers and perform literature searches, plan experiments, generate figures, review manuscripts and more. It can also run in an open loop to improve the next generation of ideas, based on the previous generation.

AI Scientist appears to be a significant breakthrough in generative AI, with its ability to perform research processes fully autonomously, suggesting that Sakana AI's team has achieved a level of creativity and reasoning much closer to that of humans. The implications of such a system are significant as it could potentially pave the way for continuous scientific research and overcome the limitations of human researchers such as taking coffee breaks, eating lunch and going home at the end of the day. It can lead to faster progress in various fields such as materials research, drug discovery and climate change mitigation.

However, it must be said that there is still a lot of work to be done before AI Scientist can compete on an equal footing with real human scientists. Right now, the model is limited to only one specific area of research, namely AI software development, and there may also be hardware limitations.

Furthermore, AI Scientist still needs an existing codebase as a starting point for its research, raising questions about its ability to truly innovate and develop new ideas.

Sakana AI's team also pointed out that AI Scientist has no computer vision capabilities, limiting its ability to fix visual issues in the articles it produces. It can also be prone to AI hallucinations, meaning it can sometimes implement ideas incorrectly or make unreasonable comparisons to baseline results, contaminating its results. They also admitted that the model sometimes makes "critical errors when writing and evaluating results", especially when comparing the magnitude ratio of two numbers.

While it is clear that AI Scientist demonstrates an impressive ability to innovate on existing ideas, it remains to be seen whether the system will ever be able to propose and describe truly original and paradigm-shifting ideas. AI systems are excellent at processing large amounts of data and identifying patterns within it, but it is not clear whether they will ever match the level of human intuition, creativity, and judgment needed to steer scientific inquiry toward meaningful and beneficial outcomes .