Skip to content
  • ✓ AIP1 Isambard-AI Phase 1 supported
  • ✗ AIP2 Isambard-AI Phase 2 unsupported
  • ✗ I3 Isambard 3 unsupported
  • ✗ BC5 BlueCrystal 5 unsupported

Kubernetes: Gateway Service - Experimental Access Only

Warning

This guide describes the experimental Kubernetes service on Isambard-AI Phase 1, which is invite-only and not designed for production workloads. See the Kubernetes overview for details of the service, the cluster and the policies that apply to it.

The Kubernetes Gateway Service makes a model served by a KServe LLMInferenceService reachable from outside Isambard-AI, for example from your laptop or from a coding agent such as opencode. Requests are authenticated with an access token, which you get using the Beacon CLI.

How the Gateway Service works

  • brics-gateway is an LLMInferenceServiceConfig provided by BriCS. When an LLMInferenceService lists it in baseRefs, KServe creates an HTTPRoute that attaches the model to the cluster's gateway (ingress-gateway in the envoy-gateway-system namespace).
  • Only LLMInferenceServices can create HTTPRoutes on the gateway at the moment.
  • When an HTTPRoute is created, a security policy is generated for it. The policy only allows requests from the user who created the LLMInferenceService. Other members of your project cannot use your model.

Sharing a model with your project

We may add an option to let all members of a project reach a model in a future release.

Attach an LLMInferenceService to the Gateway Service

Add brics-gateway to baseRefs and set router.route to {} in your LLMInferenceService:

spec:
  baseRefs:
    - name: brics-gateway

See LLM inference with KServe for a complete example, which already includes these settings.

The model is then available at:

https://apps.isambard.ac.uk/k8s-gateway/<PROJECT>/<LLMISVC_NAME>/

where <PROJECT> is your project's short code and LLMISVC_NAME is the name of your LLMInferenceService.

Custom routes

If you define your own route in router.route rather than using {}, every path in it must start with /<PROJECT>, where <PROJECT> is your project's short code.

To check the HTTPRoute that KServe created, and the paths it serves, you can view the LLMInferenceService resources in your namespace:

kubectl get llmisvc -n <PROJECT>

Install Beacon

Install Beacon on the machine you will send requests from.

  1. Download the binary for your platform from the Beacon releases page. For example, on Linux x86_64:

    curl --location --output beacon https://github.com/isambard-sc/beacon/releases/latest/download/beacon-linux-x86_64

    For other platforms, replace beacon-linux-x86_64 with the file for your platform, such as beacon-macos-aarch64 for macOS on Apple silicon. See the Beacon README for the full list.

  2. Make it executable and put it on your PATH:

    chmod +x beacon
mv beacon ~/.local/bin/

Get an access token

  1. Log in:

    beacon auth

    Beacon prints a URL. Open it in your browser and log in through Keycloak with your Identity Provider.

  2. Print the access token:

    beacon token

The token can't be refreshed. When it expires, beacon token fails and you need to run beacon auth again.

To check who you are logged in as, or to remove the cached token:

beacon whoami
beacon logout

Note

The Beacon token is only for the Gateway Service. It is separate from the token kubectl uses to manage resources in the cluster.

Send requests to the model

The Gateway Service exposes the vLLM OpenAI-compatible API. Pass the token in the Authorization header. In the following examples, replace <PROJECT> with your project's short code, and gpt-oss-120b-2n8g-llmisvc with the name of your LLMInferenceService if you changed it.

To list the models being served:

curl https://apps.isambard.ac.uk/k8s-gateway/<PROJECT>/gpt-oss-120b-2n8g-llmisvc/v1/models \
  --header "Authorization: Bearer $(beacon token)"

To send a chat completion request:

curl https://apps.isambard.ac.uk/k8s-gateway/<PROJECT>/gpt-oss-120b-2n8g-llmisvc/v1/chat/completions \
  --header "Authorization: Bearer $(beacon token)" \
  --header "Content-Type: application/json" \
  --data '{"model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "What is a supercomputer?"}]}'

The model field must match the model id returned by /v1/models, which is the spec.model.name of your LLMInferenceService.

Use the model with opencode

Any tool that supports an OpenAI-compatible API can use the model, by setting its base URL to https://apps.isambard.ac.uk/k8s-gateway/PROJECT/LLMISVC_NAME/v1 and its API key to the Beacon token. The Beacon repository has an example opencode configuration.