Creating Inference Checkpoints
Overviews how to convert your inference checkpoint for various model servers
Tag: Tutorials
GKE cross region capacity chasing with SkyPilot
In this tutorial, we will demonstrate how to leverage the open-source software SkyPilot to help GKE customers efficiently obtain accelerators across regions, ensuring workload continuity and optimized resource utilization.
E2E GenAI
This guide demonstrates deploying an end-to-end Generative AI application on Google Kubernetes Engine (GKE). It utilizes a Hugging Face model with Langchain for prompt engineering, Ray Serve for model inference, a Flask API for the backend, and a React frontend for user interaction. The setup includes infrastructure provisioning with Terraform, model experimentation in Jupyter Notebook, and containerized deployment of the backend and frontend services to GKE, all managed through kubectl.
Finetuning Gemma 3-1B-it on L4
This tutorial guides you through fine-tuning the Gemma 3-1B-it language model on Google Kubernetes Engine (GKE) using L4 GPU, leveraging Parameter Efficient Fine Tuning (PEFT) and LoRA. It covers setting up a GKE cluster, containerizing the fine-tuning code, running the fine-tuning job, and uploading the resulting model to Hugging Face. Finally, it demonstrates how to deploy and interact with the fine-tuned model using vLLM on GKE.
Running Flyte on GKE
This guide illustrates the deployment of Flyte on Google Kubernetes Engine (GKE) using Helm, utilizing Google Cloud Storage for scalable data storage and Cloud SQL PostgreSQL for a reliable metadata store. By the end of this tutorial, you will have a fully functional Flyte instance on GKE, offering businesses seamless integration with the GCP ecosystem, improved resource efficiency, and cost-effectiveness.
Hugging Face TGI
This guide demonstrates how to deploy a Hugging Face Text Generation Inference (TGI) server on Google Kubernetes Engine (GKE) using NVIDIA L4 GPUs, enabling you to serve large language models like Mistral-7b-instruct. It walks you through creating a GKE cluster, deploying the TGI application, sending prompts to the model, and monitoring the service’s performance using metrics, while also providing instructions for cleaning up the cluster.
Inference servers
Deploying and managing servers dedicated to performing inference tasks for machine learning models.
Serving LLM with TPUs using Jetstream
This tutorial shows you how to serve a large language model (LLM) using Tensor Processing Units (TPUs) on Google Kubernetes Engine (GKE) with JetStream and MaxText.
Kserve
This tutorial will guide you step by step through the process of installing KServe in a GKE Autopilot cluster.
Deploying a Persistent Chatbot on Google Cloud Platform with LangChain, Streamlit, and IAP
In this tutorial, you will learn how to deploy a chatbot application using LangChain and Streamlit on Google Cloud Platform (GCP).
Fine-Tuning Gemma 2-9B on GKE using Metaflow and Argo Workflows
This tutorial will provide instructions on how to deploy and use the Metaflow framework on GKE (Google Kubernetes Engine) and operate AI/ML workloads using Argo-Workflows.
Fine-tune gemma-2-9b and track as an experiment in MLFlow
In this tutorial we will fine-tune gemma-2-9b using LoRA as an experiment in MLFlow. We will deploy MLFlow on a GKE cluster and set up MLFlow to store artifacts inside a GCS bucket. In the end, we will deploy a fine-tuned model using KServe.
Efficient GPU Resource Management for ML Workloads using SkyPilot, Kueue on GKE
This tutorial expands on the SkyPilot Tutorial by leveraging Dynamic Workload Scheduler with the help of an open-source project called Kueue
Skypilot
In this tutorial, we will demonstrate how to leverage the open-source software SkyPilot to help GKE customers efficiently obtain accelerators across regions, ensuring workload continuity and optimized resource utilization.
vLLM GPU/TPU Fungibility
This tutorial shows you who to serve a large language model (LLM) using both Tensor Processing Units (TPUs) and GPUs on Google Kubernetes Engine (GKE) using the same deployment with vLLM