Self-Supervised Learning for the Real World
Spearheaded by advances in natural language processing, machine learning is undergoing a transformative shift towards large, generalist models trained with self-supervised learning (SSL). In this talk, I’ll discuss two challenges lying ahead for this paradigm, as well as some paths towards surmounting them.
First, I’ll discuss the problem of task ambiguity. While the space of tasks that models can perform is expanding rapidly, the number of bits used to specify the task is shrinking. Given these two opposing forces, how do we ensure that models learn the tasks we intend? I’ll discuss how we can measure the effects of such task ambiguity on humans and language models, as well as work showing how two-way interaction between users and large models can make strides on this problem in NLP and computer vision.
Second, I’ll discuss the challenge of domain-agnostic SSL, necessary for realizing the benefits of SSL in high-impact settings such as healthcare, the sciences, and engineering. I’ll present DABS, a novel kind of Domain-Agnostic Benchmark for SSL algorithms, covering data from 12 different fields (e.g., text, genomics, wearable sensors, and particle physics). With DABS, we develop and present the first SSL methods which succeed on such a broad range of modalities.