Are protein language models the new universal key?

Abstract

Protein language models (pLMs) capture some aspects of the grammar of the language of life as written in protein sequences. The so-called pLM embeddings implicitly contain this information. Therefore, embeddings can serve as the exclusive input into downstream supervised methods for protein prediction. Over the last 33 years, evolutionary information extracted through simple averaging for specific protein families from multiple sequence alignments (MSAs) has been the most successful universal key to the success of protein prediction. For many applications, MSA-free pLM-based predictions now have become significantly more accurate. The reason for this is often a combination of two aspects. Firstly, embeddings condense the grammar so efficiently that downstream prediction methods succeed with small models, i.e., they need few free parameters in particular in the era of exploding deep neural networks. Secondly, pLM-based methods provide protein-specific solutions. As additional benefit, once the pLM pre-training is complete, pLM-based solutions tend to consume much fewer resources than MSA-based solutions. In fact, we appeal to the community to rather optimize foundation models than to retrain new ones and to evolve incentives for solutions that require fewer resources even at some loss in accuracy. Although pLMs have not, yet, succeeded to entirely replace the body of solutions developed over three decades, they clearly are rapidly advancing as the universal key for protein prediction.

Topics & Concepts