Tutorial: GPU-Accelerated Serverless Inference With Google Cloud Run

Tutorial: GPU-Accelerated Serverless Inference With Google Cloud Run

Recently, Google Cloud launched GPU support for the Cloud Run serverless platform. This feature enables developers to accelerate serverless inference of models deployed on Cloud Run. In this tutorial, I will walk you through the steps of deploying Llama 3.1 Large Language Model (LLM) with 8B parameters on a GPU-based Cloud Run service. We will use the Text Generation Inference (TGI) server from Hugging Face as the model server and inference engi…