Skip to content
  • ✓ AIP1 Isambard-AI Phase 1 supported
  • ✗ AIP2 Isambard-AI Phase 2 unsupported
  • ✗ I3 Isambard 3 unsupported
  • ✗ BC5 BlueCrystal 5 unsupported

Kubernetes - Experimental Access Only

Warning

This section describes the experimental Kubernetes service on Isambard-AI Phase 1. Please note that this service is experimental, under active development and not designed for production workloads.

Access must be requested

At this time, access to this service is on an invite-only basis. Being a member of an Isambard-AI project does not give you access to the Kubernetes service. Users must be manually added to the service by the BriCS team. If you have not been added, the steps in these guides will not work for you.

Introduction

We are building Kubernetes as a Service for Isambard-AI, so that users can run containerised workloads with the same flexibility they would get from a modern cloud platform.

The service currently provides:

  • Argo Workflows - a web interface for running container-based workflows, including ready-made templates for deploying LLMs.
  • kubectl access - direct access to your project's namespace from your own machine, using kubectl.
  • LLM inference with KServe - serving LLMs with a KServe LLMInferenceService.
  • The Kubernetes Gateway Service - making a model served by an LLMInferenceService reachable from outside Isambard-AI, using the Beacon CLI to authenticate.

Cluster information

The Kubernetes cluster on Isambard-AI Phase 1 consists of 24 GH200 nodes.

Namespaces and service accounts

Each project is assigned a dedicated namespace, with the project short code as its name. You are only able to create resources within your project's namespace.

A single service account, PROJECT-USERNAME, is provisioned for you in each project, where PROJECT is your project's namespace and USERNAME is your username. Argo Workflows run as this service account. When you log in with kubectl, you are granted the same permissions, through the same role binding, as this service account.

Storage

Storage is provided by node-local storage through the Longhorn StorageClass. Lustre storage is not available at this time.

To create a PersistentVolumeClaim on Longhorn:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: example-pvc
spec:
    accessModes:
        - ReadWriteOnce
    storageClassName: longhorn
    resources:
        requests:
            storage: 10Gi

To use it in your pod spec:

template:
    volumes:
        - name: data
          persistentVolumeClaim:
              claimName: example-pvc
    containers:
        volumeMounts:
            - name: data
              mountPath: /data

GPUs

To request GPUs, and CXI devices for RDMA between nodes, you use a ResourceClaim. A ResourceClaim requests a pre-existing ResourceClaimTemplate. The following ResourceClaimTemplates are available to use.

  • gh200-x1 - requests one GH200 GPU.
  • gh200-x4 - requests all four GH200s on one node.
  • gh200-x1-cxi-x1 - requests one GH200 and one CXI device that share a NUMA node.
  • gh200-x4-cxi-x4 - requests all four GH200s on one node, along with all four CXI devices.

Subject to change

These ResourceClaimTemplates are subject to change as the service develops.

To make a resource claim in your pod spec:

template:
    resourceClaims:
        - name: whole-node
          resourceClaimTemplateName: gh200-x4-cxi-x4
    containers:
        resources:
            claims:
                - name: whole-node

Modules

We have developed k8s-modules, which provides an experience similar to environment modules for Kubernetes pods. You load a module by adding an annotation to a pod's metadata. When the pod is created, the module injects the required libraries and environment variables into it. These changes are visible in the pod spec once the pod has been created.

The following modules are available:

  • aws-ofi-nccl - provides the aws-ofi-nccl NCCL network plugin, which enables RDMA over the Slingshot network. Use it with a ResourceClaimTemplate that includes CXI devices, such as gh200-x4-cxi-x4.

More modules will be added over time.

To load a module, add the modules.isambard.ac.uk/load annotation to your pod's metadata:

metadata:
  annotations:
    modules.isambard.ac.uk/load: aws-ofi-nccl

In an Argo Workflow, set this annotation in the metadata field of the template that creates the pod.

Policies

The following policies have been configured, and are enforced:

  • A resource reaper automatically terminates all pods and owning-resources, such as deployments, after 24 hours.
  • Cross-namespace network traffic is blocked.
  • Only the IPC_LOCK capability is permitted on pods.
  • All pods must set runAsNonRoot: true and runAsUser greater than 0.
  • All pod specs must contain resource requests for CPU and RAM.
  • Privileged pods are not permitted.
  • Host PID, Network, IPC are not permitted.