John Lu
John is a Senior AI Engineer, currently focused on developing NLP applications.
He is deeply motivated by challenges and tends to be excited by breaking conventional ways of thinking and doing. With prior experiences in Software Engineering, he works on combining the latest AI technology and engineering to transform challenges into practical solutions.
Sessions
Ever wonder how Vision Language Models (VLMs) works? VLMs are built on a vision encoder and a language decoder. It accepts both images and text as inputs and can answer vision-language questions with detailed insights. Building a VLM from scratch allows us to customize the component for our application. The goal of this talk is to demonstrate how VLMs could be implemented in a Pythonic way. To do so, we're going to build the PaliGemma VLM completely from scratch all using Python.
Transformer architecture can be used for various NLP and CV tasks. They are pre-trained to generate text and images based on large datasets. Building a transformer from scratch allows us to customize the component for our application. The goal of this talk is to demonstrate how the transformer model could be implemented on JAX. To do so, we're going to build a general purpose transformer completely from scratch all with JAX.