I recently built an MCP tool in k8s_infra/plugins/infra-observer for AI-assisted infrastructure triage. It is not a wrapper around a remote terminal, and it does not hand kubectl to a model. Its job is narrower: give the model enough information to investigate a problem without letting it change the environment.
The reason is straightforward. Once an AI can SSH to a host and pass arbitrary commands, it has been given too much discretion. Calling a tool read-only does not prevent it from reading a Secret, running something in a Pod, or following an instruction hidden in a log. What I need from it is much less: tell me whether Pods are unhealthy, where a rollout is stuck, whether Argo CD is in sync, whether a Service has endpoints, and whether a service is showing errors in OTel.
Start by making the tool surface smaller
infra-observer has no shell, generic SSH, or generic kubectl tool. It has named operations for the things I check most often: a health summary, unhealthy Pods, rollout status, events, bounded Pod logs, Argo CD status, Service endpoints, Docker status, and an OTel error summary for one service.
That is less flexible than exposing a shell, but the inputs are clear. Namespaces, workloads, and service names must be valid Kubernetes names. Log lines, query limits, output size, and timeouts are all bounded. The local client builds argv arrays directly, without a shell. There is simply no tool for Secret reads, Pod exec, port forwarding, resource mutation, arbitrary URLs, or arbitrary PromQL, LogQL, and TraceQL.
Kubernetes and observability are different data sources, but I apply the same approach to both. Kubernetes uses a separate infra-observer identity with least-privilege RBAC. The tool can also use kubectl auth can-i to verify that Secret, exec, mutation, and impersonation access are denied. On the OTel side, it only checks fixed Grafana, Loki, Tempo, Prometheus, Vault, and Collector endpoints, and fetches error-related information for one validated service name. The model gets a lead for investigation, not a pass to browse the whole platform.
The SSH host does not trust the client either
A local allowlist is not enough. A bypassed plugin or SSH configuration could still send an unrestricted command to the host. I use a dedicated SSH key with a forced command in authorized_keys, so a connection reaches a root-owned forced_command.py instead of a shell.
This is a shortened version of one of its kubectl rules. It compares the complete argv rather than checking only the start of a string. The namespace must still be a valid DNS label and the output format is fixed. Commands that have no explicit rule return False.
def kubectl_allowed(args: list[str]) -> bool:
if tuple(args) in EXACT_KUBECTL_COMMANDS:
return True
if len(args) == 8 and args[:4] == ["kubectl", "get", "pods", "-n"]:
return is_name(args[4]) and args[5:] == [
"-o", POD_STATUS_COLUMNS, "--no-headers"
]
# Other explicit rules cover workloads, events, logs, and endpoints.
return False
After a command passes the allowlist, the wrapper runs it with a fixed binary path, kubeconfig, and clean environment. It does not inherit the client’s HOME, PATH, KUBECONFIG, Docker, or XDG settings, and it disables TTY, port forwarding, agent forwarding, and X11. The final step is intentionally small: parse SSH_ORIGINAL_COMMAND, reject an argv that does not match a rule, then use os.execve() for an approved binary.
def main() -> None:
args = parse_original_command(os.environ.get("SSH_ORIGINAL_COMMAND", ""))
if not allowed(args):
raise SystemExit(126)
executable = executable_path(args[0])
if executable is None:
raise SystemExit(126)
os.chdir("/var/empty")
os.execve(executable, [executable, *args[1:]], EXECUTION_ENVIRONMENT)
There is an easy-to-miss deployment requirement here: the wrapper, /var/empty, and the observer kubeconfig must be owned by root and not writable by the SSH user. Otherwise the fixed paths are not a real boundary. Before issuing diagnostics, the local tool also asks the wrapper for its policy version. If it does not match, it stops rather than assuming both sides enforce the same rules.
This key is not meant to reduce the privileges of an existing administrator account. If that account can already operate Docker or the cluster, it remains a trusted operator. The constraint applies to the key used by the AI: it may perform only these reviewed reads, and must not be replaced with a general administration key.
Logs are noisy, and local models have less context
Logs and events are not trusted input. They can contain tokens, internal URLs, stack traces, or text written specifically to distract an agent. The tool caps output, makes a best-effort attempt to redact sensitive-looking values, removes control sequences, and labels the result as untrusted evidence. That does not make logs safe; it helps prevent a model from treating commands or links in a log as new instructions.
There is also a practical local-LLM concern: tool schemas and long output consume context quickly. This plugin started with 56 narrow or overlapping tools and is now down to 19. The full profile keeps the complete schema. Codex and Claude Code use balanced: all safe tools are available, while the model-facing result is capped at 4 KB. Qwen Code and LM Studio use compact, which exposes eight common first-pass tools for health, Pods, rollout, events, logs, Argo CD, and Docker, then returns roughly 2 KB of summary JSON.
This is more than truncating text at the end. Kubernetes inventory begins with a summary and a short table. Normal diagnostics are limited to 8 KB and dedicated log tools to 12 KB, then the profile reduces the payload passed to the model again. On a smaller model, getting a health summary first and drilling into one workload’s rollout, events, or logs is generally quicker and more useful than dumping the whole cluster state into context.
In practice, I start by letting the AI read the GitOps repository so it understands the desired state. Then I check the live cluster health summary. If there is a real signal, I drill into one workload or service. The AI can organize the evidence, suggest likely causes, and point to a manifest worth changing, but it does not repair anything directly. Changes still go through a Git diff, review, and Argo CD.
This tool does not turn an AI into a Kubernetes administrator. It turns the first round of manual observation into a small set of repeatable reads with a clear boundary. That has been useful enough for me.