Many sequence-only pLMs just require replacing the tokenizer with a one-hot encoder, so they’re simpler than LLMs to port. My lab has a few (unreleased - we’ll get there) protein structure transformers, where we’ve got some available code for key components if others want to build (eg. GitHub - MurrellGroup/InvariantPointAttention.jl: Julia implementation of AlphaFold 2's Invariant Point Attention and GitHub - MurrellGroup/MessagePassingIPA.jl).
6 Likes