Leveraging Pre-Trained Transformer Models for Protein Function Prediction - Tia Pope, North Carolina A&T
Presented at All Things Open AI 2025 Presented by Tia Pope - North Carolina A&T Title: Leveraging Pre-Trained Transformer Models for Protein Function Prediction Abstract: Transformer-based models, such as ProtGPT2 and ESM, are revolutionizing protein sequence analysis by enabling detailed embeddings and advanced function prediction. This talk provides a hands-on introduction to using pre-trained open-source transformer models for generating protein embeddings and leveraging them for classification tasks. Attendees will learn to tokenize sequences, extract embeddings, and implement machine-learning pipelines for protein function annotation based on Gene Ontology (GO) or Enzyme Commission (EC) numbers. This session will showcase how pre-trained transformers can democratize access to advanced protein analysis techniques while addressing scalability and explainability challenges. After the talk, the speaker will provide a notebook to test basic functionality, enabling participants to explore the concepts discussed. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen Bluesky: https://bsky.app/profile/allthingsopen.bsky.social 2025 conference: https://2025.allthingsopen.org/
Leveraging Pre-Trained Transformer Models for Protein Function Prediction - Tia Pope, North Carolina A&T
- 1. Leveraging Pre-Trained Transformer Models for Protein Function Prediction Tia Pope
- 2. I am not a biochemist, but… • Cybersecurity • Healthcare • Software Engineering • Interoperability • Machine Intelligence I computationally study proteins… Protein AI • Novel Proteins • In Silico Evaluation • 3rd Year Ph.D. Candidate • Collaborations with J&J, Lilly, MIT LL My Background
- 3. Protein Wars The Good Side – Our Body’s Defenders • Antibodies – fight off viruses and bacteria. • Enzymes – builders and repair workers • Hormones – messengers telling our body what to do. The Enemy – Bad or Sneaky Proteins • Virus Proteins – Like spies (e.g., COVID-19's spike protein) that trick our cells into letting them in. • Mutated Proteins – Sometimes, our own proteins go rogue and cause diseases (like cancer). • Misfolded Proteins – When proteins fold wrong, they can damage the brain (like in Alzheimer’s). Image: https://mutantreviewersmovies.com/2022/06/07/osmosis-jones-2001- what-you-never-wanted-to-know-about-your-own-body/
- 6. COVID-19 is a Protein? No, but uses them. • Membrane (M) protein – Helps shape the virus. • Envelope (E) protein – Plays a role in viral assembly and release. • Nucleocapsid (N) protein – Binds to the viral RNA genome. • Spike (S) protein – Helps the virus attach to and enter human cells.
- 8. Advances in BioChemistry AlphaFold revolutionized protein structure prediction by using deep learning to accurately model 3D protein structures from amino acid sequences, reducing prediction time from months or years to just hours or minutes.
- 9. There’s…
- 10. But there is also…
- 11. Feature RNN (Recurrent Neural Network) Transformer Processing Sequential (one word at a time) Parallel (all words at once) Speed Slower (can't process in parallel) Faster (processes entire input at once) Long-Range Dependencies Struggles with long sentences (information gets lost overtime) Captures long-range dependencies efficiently Memory Usage Uses less memory per step but needs more steps Requires more memory but fewer steps Training Efficiency Harder to train (vanishing gradient problem) Easier to train with large datasets Example Task Predicting the next word in a sequence Translating entire paragraphs accurately Real-World Example Older chatbots, speech recognition (e.g., Siri’s early versions) GPT, BERT, modern AI chatbots
- 12. AI for Treatments AI dramatically accelerated experimentation, testing and vaccine development. Example includes helping bring COVID-19 vaccines to the public in less than a year—a process that usually takes many years.
- 13. Protein Function Using AI Prediction Tools
- 14. Protein Production Natural biological process that we don’t 100% understand. The majority of known protein sequences lack experimental validation and are labeled as hypothetical or uncharacterized proteins. Image: https://slideplayer.com/slide/15368149/
- 15. Every BODY is Different & More to STUDY https://www.thebureauinvestigates.com/stories/2024-09-16/superbugs-will-kill-three-every-minute-by- 2050#:~:text=Published%20September%2016%202024&text=Drug%2Dresistant%20infections%20ar e%20expected,have%20become%20untreatable%20by%20antibiotics.
- 16. Protein Function Prediction Helps us understand biological processes, develop new drugs, diagnose diseases and engineer novel proteins for medicine, biotechnology and environmental applications. • Disease Research: Identifies disease-related proteins, aiding in treatments for cancer, Alzheimer’s, and viral infections (e.g., COVID-19, Superbug X). • Drug Discovery: Helps design targeted drugs by identifying protein interactions and binding sites. Or identify novel sequences and variants of existing proteins. • Synthetic Biology: Enables the design of custom proteins for bioengineering, like enzymes for sustainable fuels. • Environmental Science: Helps in breaking down pollutants or engineering bacteria for waste management.
- 17. ESM ProtGPT2 Evolutionary Scale Modeling Built on GPT-2, Not ProteinGPT
- 18. ProtGPT2 for Novel Protein Generation • Install the required libraries • Load the ProtGPT2 model from Hugging Face (Pre-Trained) • Provide a seed sequence (optional) or generate a new one • Generate a new protein sequence Pre-Trained Decoder Transformer
- 19. ProtGPT2 Example 1: • Input (Seed Sequence): "MKTLLLTLV" • Output (Generates): "MKTLLLTLVVVTIVCLDLGYTGTVNNSM...” Example 2: • Input (Unconditional or No Seed): " " • Output (Generates): "MGRKYLTVASM..." The Actual 3D Structure Predicted with ESMFold
- 20. Using ESM for Function Prediction • Install the ESM model dependencies • Load the ESM model from Hugging Face (Pre-Trained) • Prepare the test sequence • Run the function prediction • Interpret the output Example: • Input (Protein Sequence): "MKTLLLTLVVVTIVCLDLGYTGRKYLTVNNSM..." • Output (Predicted Function): Enzyme (Kinase) Gene Ontology: GO:0004674 (Protein Serine/Threonine Kinase Activity) Pre-Trained Encoder Transformer Some ESM versions have decoder capabilities
- 21. ESM (Encoder-Based) Use Cases Task Input Output Protein Folding (ESMFold) Sequence 3D structure (PDB) Mutation Effects Wild-type & mutant sequence Mutation heatmap Evolutionary Velocity Two sequences Evolutionary direction Binding Site Prediction Sequence Binding site classification (0/1) PPI Prediction Two sequences Interaction probability (0-1) AI-Guided Protein Design Protein motif Designed full sequence MSA-Free Homology Detection Sequence List of similar proteins Function Prediction Sequence Biological function label Intrinsic Dimension Analysis Sequence Complexity score Persistent Homology Clustering Protein embeddings Cluster assignments Fine-Tuning (LoRA) Labeled protein sequences Improved function prediction
- 22. Bacta Tank • Learn to Leverage Pre- Trained Transformer Models for Protein Function Prediction • Maybe we can work together to take down the next super bug… • Or create a Bacta!
- 23. Connect. Fact Check. Try it Out! Contribute Notebook & Sample Data Referenced Papers & Articles LinkedIn