• ✓ AIP1 Isambard-AI Phase 1 supported
• ✗ AIP2 Isambard-AI Phase 2 unsupported
• ✗ I3 Isambard 3 unsupported
• ✗ BC5 BlueCrystal 5 unsupported

Kubernetes: LLM Inference with KServe - Experimental Access Only

Warning

This guide describes the experimental Kubernetes service on Isambard-AI Phase 1, which is invite-only and not designed for production workloads. See the Kubernetes overview for details of the service, the cluster and the policies that apply to it.

This guide shows how to serve an LLM with a KServe LLMInferenceService using kubectl. The example deploys one instance of OpenAI's GPT-OSS-120B model on eight GH200s across two nodes.

A multi-node example, not a sensible deployment

Spreading GPT-OSS-120B over eight GPUs and two nodes is not a sensible way to serve this model. A single GPU would be fine, and would be simpler and faster to start. The example is deliberately oversized to show you how to deploy a model across multiple nodes, which you will need for models too large for one node.

You need kubectl access to follow this guide. If you would rather use a web interface, the Argo Workflows gpt-oss-120b-2n8g-llmisvc template performs the same steps.

About LLMInferenceService

The LLMInferenceService provides a Custom Resource Definition (CRD) to serve generative AI models. It is built on top of llm-d and combines vLLM's inference engine with Kubernetes orchestration and intelligent routing capabilities. It supports KV-cache aware scheduling, disaggregated prefill-decode serving, and distributed inference. All of the required configuration for a model, such as parallelism settings, disaggregated prefill-decode, routing and scheduling, is directly configurable in the LLMInferenceService CRD. See our Distributed vLLM Inference tutorial for background on tensor and pipeline parallelism.

Some configuration settings, such as parallelism, only apply when using the built-in llm-d container. If you provide your own container image, using an NGC image for example, these settings are not automatically picked up and you need to pass the equivalent options to vLLM yourself.

KServe handles model selection within the model spec of the LLMInferenceService. You can specify a path to a model in a PVC or the Hugging Face URI for a model. This is then automatically mounted at /mnt/models in the containers created in the LLMInferenceService pods.

Download the model into a PVC

Downloading the model into a PersistentVolumeClaim (PVC) once means it does not need to be downloaded again each time you deploy it.

1. Create a PVC to hold the model. Save the following as hf-model-cache.yaml:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hf-model-cache
   spec:
     accessModes:
       - ReadWriteMany
     storageClassName: longhorn
     resources:
       requests:
         storage: 250Gi
   

   The PVC uses ReadWriteMany so that the pods on both nodes can mount it at the same time. Then create it:

   kubectl apply --filename hf-model-cache.yaml
   

2. Create a pod to download the model and the Tiktoken encodings that GPT-OSS needs into the PVC. Save the following as download-gpt-oss-120b.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: download-gpt-oss-120b
   spec:
     restartPolicy: Never
     securityContext:
       runAsNonRoot: true
       runAsUser: 1000
       fsGroup: 1000
     volumes:
       - name: models
         persistentVolumeClaim:
           claimName: hf-model-cache
     containers:
       - name: download
         image: python:3.12-slim
         env:
           - name: HOME
             value: /tmp
         resources:
           requests:
             cpu: "2"
             memory: 4Gi
         volumeMounts:
           - name: models
             mountPath: /models
         command: ["bash", "-c"]
         args:
           - |
             set -o errexit
             pip install --quiet --user huggingface_hub
             ~/.local/bin/hf download openai/gpt-oss-120b \
               --local-dir /models/gpt-oss-120b \
               --exclude "original/*" --exclude "metal/*"
             mkdir -p /models/gpt-oss-120b/tiktoken
             for encoding in o200k_base cl100k_base; do
               python -c "import urllib.request; urllib.request.urlretrieve('https://openaipublic.blob.core.windows.net/encodings/${encoding}.tiktoken', '/models/gpt-oss-120b/tiktoken/${encoding}.tiktoken')"
             done
   

   The pod meets the cluster's policies: it runs as a non-root user and sets CPU and memory requests. fsGroup makes the PVC writable by that user. The --exclude options skip copies of the weights in other formats that vLLM does not need.

   Gated models

   To download a gated model from Hugging Face, create a Kubernetes Secret containing your Hugging Face token and expose it to the container as the HF_TOKEN environment variable.

   Create the pod and follow its progress:

   kubectl apply --filename download-gpt-oss-120b.yaml
   kubectl logs --follow download-gpt-oss-120b
   

3. Check that the download finished successfully:

   kubectl get pod download-gpt-oss-120b
   

   The STATUS should be Completed before you continue. If it is Error, check the output of kubectl logs download-gpt-oss-120b before deleting the pod. If it is still running, wait for it to finish and check again.

   Once the download has completed successfully, delete the pod:

   kubectl delete pod download-gpt-oss-120b
   

Deploy the LLMInferenceService

1. Save the following as gpt-oss-120b-llmisvc.yaml, replacing PROJECT with your project's short code:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: gpt-oss-120b-2n8g-llmisvc
     namespace: PROJECT
   spec:
     annotations:
       modules.isambard.ac.uk/load: aws-ofi-nccl
     baseRefs:
       - name: brics-gateway
     model:
       name: openai/gpt-oss-120b
       uri: pvc://hf-model-cache/gpt-oss-120b
     replicas: 1
     parallelism:
       tensor: 4
       pipeline: 2
     template:
       resourceClaims:
         - name: whole-node
           resourceClaimTemplateName: gh200-x4-cxi-x4
       containers:
         - name: main
           resources:
             requests: {cpu: "72", memory: 64Gi, ephemeral-storage: 20Gi}
             limits:   {cpu: "72", memory: 64Gi, ephemeral-storage: 20Gi}
             claims:
               - name: whole-node
           env:
             - name: TIKTOKEN_ENCODINGS_BASE
               value: /mnt/models/tiktoken
             - name: HF_HUB_CACHE
               value: /mnt/models
     worker:
       resourceClaims:
         - name: whole-node
           resourceClaimTemplateName: gh200-x4-cxi-x4
       containers:
         - name: main
           resources:
             requests: {cpu: "72", memory: 64Gi, ephemeral-storage: 20Gi}
             limits:   {cpu: "72", memory: 64Gi, ephemeral-storage: 20Gi}
             claims:
               - name: whole-node
           env:
             - name: TIKTOKEN_ENCODINGS_BASE
               value: /mnt/models/tiktoken
             - name: HF_HUB_CACHE
               value: /mnt/models
   

   In this manifest:

   • parallelism splits the model across four GPUs on each of two nodes: tensor parallelism within a node and pipeline parallelism across nodes.
   • template describes the pod on the first node and worker describes the pod on the second. Each claims a whole node, with all four GPUs and CXI devices, through the gh200-x4-cxi-x4 ResourceClaimTemplate.
   • The aws-ofi-nccl module enables RDMA over the Slingshot network between the two nodes.
   • baseRefs and router.route attach the model to the Kubernetes Gateway Service. See Kubernetes Gateway Service. Remove them if you do not need to reach the model from outside the cluster.

2. Create the LLMInferenceService:

   kubectl apply --filename gpt-oss-120b-llmisvc.yaml
   

3. Wait for the model to load. This can take several minutes:

   kubectl wait --for=condition=Ready llminferenceservice/gpt-oss-120b-2n8g-llmisvc --timeout=1h
   

   To see the pods and their progress:

   kubectl get llminferenceservice
   kubectl get pods
   kubectl logs --follow POD_NAME
   

   Debugging multi-node communication

   If the model fails to load across the two nodes, add the environment variables NCCL_DEBUG=INFO and FI_LOG_LEVEL=DEBUG to both containers to get more detailed logs from NCCL and libfabric.

Once the LLMInferenceService is ready, you can send requests to it through the Kubernetes Gateway Service.

Delete the LLMInferenceService

The resource reaper deletes the LLMInferenceService after 24 hours. To free the GPUs sooner, delete it yourself when you have finished:

kubectl delete llminferenceservice gpt-oss-120b-2n8g-llmisvc

The model stays in the hf-model-cache PVC, so you can deploy it again without downloading it. To delete the PVC and the model:

kubectl delete pvc hf-model-cache