Tel Aviv University study shows AI can learn without real-world data

Credit: For illustration only

An AI model can develop general prediction skills without training on any real-world data, according to a new study by researchers from Tel Aviv University and Stanford University. Instead of learning from text, images and recordings, two models learned from each other: one created exercises and the other learned to solve them.

The research was led by Dr. Yoav Levine of Tel Aviv University’s Blavatnik School of Computer Science and AI, as part of the postdoctoral work of Dr. Kfir Dolev.

In the training method the team developed, both models start from scratch. The “teacher” writes short computer programs that generate artificial sequences of symbols, and the “student” practices predicting the next symbol in each sequence. The teacher improves too: it gets feedback on how much its exercises helped and adjusts their difficulty to the student’s progress.

After training, the researchers tested the student model on text, images, sounds, music, code and DNA sequences it had never seen. The more computing power went into training, the better its predictions became across many of these domains. It was also able to infer rules from examples shown to it during testing, without further training. Pretraining with the new method also helped the model learn faster when it was later exposed to real data.

Familiar mathematical patterns emerged in the material the system generated for itself, including the Fibonacci sequence, in which each number is the sum of the two before it. The system was never given the sequence. It produced structured sequences the second model could learn from.

The method has limits. It does not teach a model facts about the world without exposure to it, but may help it build skills to learn from the world more efficiently. The study is a proof of concept tested on small models.

“For years, the prevailing assumption was that for an AI model to learn anything useful, it first had to be fed large amounts of data from the world,” Levine said. “In this study we refuted this basic assumption of the field and showed that a model can acquire general prediction skills even when it trains only on data generated within the learning process itself.”

More information is available on the Tel Aviv University website.

We aim to provide accurate, high-quality content only.
If you spotted inaccurate or incorrect information – we want to know. Please contact us and we’ll fix the issue promptly.

More articles you might find interesting: