I will present how protein language models (PLMs) learn useful representations from large, unlabeled collections of amino-acid sequences, providing a general framework for predicting protein properties without task-specific biological features. I will trace our efforts in developing transformer-based protein modeling, from protein-family classification and prediction of protein-protein interactions to prediction of disordered regions, to current efforts to determine what PLMs know about protein fitness and epistasis.

I will show how contextual sequence representations enable accurate annotation of intrinsically disordered regions through DR-BERT, a compact model trained without explicit evolutionary or biophysical inputs. I will then focus on recent work that interrogates PLM representations to learn the fitness effects of mutations. I will demonstrate that zero-shot PLM scores correlate with epistatic patterns in experimental deep mutational-scanning data: raw scores primarily reflect local structural contacts, whereas nonlinear calibration to the experimental fitness scale exposes long-range couplings enriched in functional protein regions.

Together, these studies frame PLMs not only as flexible predictors, but also as tools for asking how sequence statistics encode protein structure, disorder, function, and evolutionary constraints.