PhD student in electrical engineering and computer science Mark Hamilton, is leading this project with colleagues at MIT’s Computer Science and Artificial Intelligence Laboratory. They aim to use machines to decipher animal communication, starting with human language acquisition.
The inspiration for this new algorithm is a movie. In one scene, the penguin falls to the ground and moans as he tries to get up. Observing that this groan seems to imply a word, Hamilton considers the idea that audio and video could be used together to teach an algorithm a language.
This idea led to DenseAV, a model designed to learn language by predicting visual content from audio. For example, hearing “bake the cake at 350” will cause the model to expect an image of a cake or oven.
But to enable audio-video matching across millions of videos, DenseAV needs to learn the context in which people are discussing. After training DenseAV on this matching task, the research team examined which pixels the model focused on when processing sounds.
When the word “dog” was spoken, the algorithm searched for images of dogs in the video stream, showing that it understood the meaning of the word. Similarly, when he heard a dog barking, he looked for the dogs in the video.
The team was wondering if DenseAV could distinguish between the word “dog” and the sound of a dog barking. By applying a dual-brain approach to DenseAV, they discovered that one side naturally focuses on language, such as the word “dog,” while the other side focuses on sounds, such as barking.
The team faced a challenging task in learning languages without text input, as they aimed to rediscover the essence of language from scratch without using pre-trained language models. This method is inspired by how children learn language by observing and listening to their environment.
Source
https://www.techtimes.com/articles/305612/20240612/mit-unveils-new-algorithm-learns-language-watching-videos.htm
https://news.mit.edu/2024/denseav-algorithm-discovers-language-just-watching-videos-0611
Share via Email
This is titled mail it to your friend.
This news our mobile application Download using
You can read it whenever you want (even offline):
Source link: https://www.donanimhaber.com/bu-algoritma-yalnizca-video-izleyerek-dil-ogrenebiliyor–178479
Web sitemizde ziyaretçilerimize daha iyi hizmet sağlayabilmek adına bazı çerezler kullanıyoruz. Web sitemizi kullanmaya devam ederseniz çerezleri kabul etmiş sayılırsınız.
Gizlilik Politikası