My Notes

Cilium and Talos Kubernetes Troubleshooting Guide

4 min read

This guide provides troubleshooting steps for common issues with Cilium networking and Talos Linux Kubernetes clusters.

1. Checking Cilium Status

Always start by checking the current status of your Cilium installation:

cilium status

Look for any reported errors and the status of key components:

  • Cilium
  • Operator
  • Envoy DaemonSet
  • Hubble Relay
  • ClusterMesh

2. TLS and Certificate Issues

2.1. Pending CSRs

Certificate Signing Requests (CSRs) must be approved for proper communication between Kubernetes components.

Check for pending CSRs:

kubectl get csr | grep Pending

Approve all pending CSRs:

kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

Verify approvals:

kubectl get csr

2.2. Regenerating Certificates in Talos

If you encounter TLS internal errors:

Check certificate status:

talosctl get certs --nodes <node-ip>

Restart kubelet to trigger certificate regeneration:

talosctl service kubelet restart --nodes <node-ip>

For a more comprehensive certificate reset:

talosctl reset --graceful=true --system-labels-to-wipe k8s.io --nodes <node-ip>

Force regenerate kubeconfig:

talosctl kubeconfig --nodes <node-ip> -f

In-place upgrade to recreate certificates:

talosctl upgrade --preserve --nodes <node-ip> --image ghcr.io/siderolabs/installer:latest

2.3. Disabling TLS (Temporary Troubleshooting Only)

Using Helm to disable Hubble TLS:

cat > cilium-values.yaml << EOF
hubble:
  tls:
    enabled: false
EOF
 
helm upgrade cilium cilium/cilium --namespace kube-system -f cilium-values.yaml

Or modify the Cilium ConfigMap directly:

kubectl edit configmap -n kube-system cilium-config
# For Hubble TLS specifically, look for and modify:
# hubble-tls-cert: ""
# hubble-tls-key: ""
# hubble-ca-cert: ""

Then restart Cilium:

kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl rollout restart daemonset/cilium -n kube-system

⚠️ Security Warning: Disabling TLS removes a critical security layer. Only use temporarily for troubleshooting.

3. Resolving Cilium-Specific Issues

3.1. Cilium Component Troubleshooting

Check Cilium pods:

kubectl get pods -n kube-system | grep cilium

View Cilium logs:

kubectl logs -n kube-system -l k8s-app=cilium
kubectl logs -n kube-system -l io.cilium/app=operator

Restart Cilium DaemonSet:

kubectl rollout restart ds/cilium -n kube-system

Restart Cilium operator:

kubectl rollout restart deployment/cilium-operator -n kube-system

3.2. Cleaning Up Failed Pods

Remove failed cilium-operator pods:

# First, identify failed pods
kubectl get pods -n kube-system | grep cilium-operator | grep -v Running
 
# Then delete them
kubectl delete pods -n kube-system $(kubectl get pods -n kube-system | grep cilium-operator | grep -v Running | awk '{print $1}')
 
# Alternative safer approach (if above doesn't work)
for pod in $(kubectl get pods -n kube-system | grep cilium-operator | grep -v Running | awk '{print $1}'); do
  kubectl delete pod -n kube-system $pod;
done

3.3. Reinstalling Cilium

If you need to reinstall Cilium completely:

# Remove existing installation
helm uninstall cilium -n kube-system
 
# Reinstall with Helm
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --version 1.17.2 --namespace kube-system

4. Debugging Network Connectivity

4.1. Check Node Communication

Verify node internal IPs:

kubectl get nodes -o wide
# Or for more detail
kubectl describe nodes | grep InternalIP

Test connectivity between nodes:

# From one node to another (using talosctl)
talosctl --nodes <node-ip> containers
talosctl --nodes <node-ip> exec --stdin /bin/sh -- ping <destination-node-ip>
 
# Test specific port connectivity (using debug pod)
kubectl run -it --rm debugger --image=nicolaka/netshoot -- bash
# Then inside the container
nc -zv <node-internal-ip> 10250

4.2. DNS and Service Connectivity

Test DNS resolution:

kubectl run -it --rm debug --image=busybox -- nslookup kubernetes.default

Check kube-proxy status:

kubectl get pods -n kube-system | grep kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy

5. Talos-Specific Troubleshooting

5.1. Get Talos System Information

View Talos services:

talosctl services --nodes <node-ip>

Check Talos logs:

talosctl logs controller-runtime --nodes <node-ip>

5.2. Kubelet Status and Configuration

Check kubelet status:

talosctl service kubelet status --nodes <node-ip>

View kubelet logs:

talosctl logs kubelet --nodes <node-ip>

5.3. Node Recovery

For more serious issues, you can recover a Talos node:

# Safe reset preserving data
talosctl reset --graceful=true --reboot --system-labels-to-wipe k8s.io --nodes <node-ip>
 
# Full reset (use with caution)
talosctl reset --graceful=false --reboot --system-labels-to-wipe all --nodes <node-ip>

6. General Kubernetes Debugging

Check system namespace:

kubectl get pods -n kube-system

View node conditions:

kubectl describe nodes

Check events:

kubectl get events --sort-by='.metadata.creationTimestamp'

7. Common Error Messages and Solutions

”remote error: tls: internal error”

  • Approve pending CSRs
  • Regenerate certificates
  • Restart kubelet
  • Check for certificate expiration

”NodeShutdown - Pod was rejected”

  • Clean up failed pods
  • Check node status with kubectl get nodes
  • Restart affected deployments
  • The pods might have failed during node maintenance - this is expected behavior

”error fetching pod status”

  • Check kubelet is running with talosctl service kubelet status --nodes <node-ip>
  • Verify API server connectivity
  • Regenerate certificates
  • Check for pending CSRs

”unable to retrieve cilium status: command failed”

  • Check cilium agent pods with kubectl get pods -n kube-system -l k8s-app=cilium
  • View cilium logs with kubectl logs -n kube-system -l k8s-app=cilium
  • Restart cilium with kubectl rollout restart ds/cilium -n kube-system

Remember to always back up your configuration before making significant changes, and consider the security implications of any troubleshooting steps you take.

Graph View