Troubleshooting Guide: GitHub Actions Runner DNS Failures

Problem Description

Symptoms:

Multiple GitHub Actions runners stuck in “Failed” state for days
Error message: “Pod has failed to start more than 5 times”
Runners cannot communicate with GitHub API
New runners immediately fail upon creation

Root Cause: DNS resolution failure preventing the GitHub Actions Runner Controller from communicating with GitHub’s API due to upstream DNS server timeouts.

Diagnostic Process

Step 1: Identify the Failed Runners

# Check the status of ephemeral runners
kubectl get ephemeralrunner -n gh-actions-runners-set -o wide

Expected Output:

NAME                                                         STATUS   MESSAGE
mktech224-github-actions-runner-generic-wkdwh-runner-dt9v4   Failed   Pod has failed to start more than 5 times
mktech224-github-actions-runner-generic-wkdwh-runner-pgn87   Failed   Pod has failed to start more than 5 times
...

Step 2: Examine Runner Details

# Get detailed information about a failed runner
kubectl describe ephemeralrunner <runner-name> -n gh-actions-runners-set

Key Information to Look For:

Phase: Failed
Reason: TooManyPodFailures
Message indicates pod startup failures

Step 3: Find the Runner Controller

# Locate the GitHub Actions runner controller pod
kubectl get pods -A -l app.kubernetes.io/name=gha-rs-controller

Step 4: Check Controller Logs for Root Cause

# Examine controller logs for errors
kubectl logs <controller-pod-name> -n <controller-namespace> --tail=100

Critical Error Pattern to Look For:

dial tcp: lookup api.github.com on 10.96.0.10:53: server misbehaving
Post "https://api.github.com/app/installations/62603260/access_tokens": giving up after 5 attempt(s)

Step 5: Investigate DNS Issues

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
 
# Examine CoreDNS logs for DNS resolution failures
kubectl logs <coredns-pod-name> -n kube-system --tail=50

DNS Error Pattern:

[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.2.72:45030->169.254.116.108:53: i/o timeout

Resolution Steps

Step 1: Force Delete Failed Runners

Since the controller cannot clean up failed runners due to DNS issues, manually remove them:

# List all failed runners
kubectl get ephemeralrunner -n gh-actions-runners-set
 
# Force delete each failed runner
kubectl delete ephemeralrunner <runner-name> -n gh-actions-runners-set --force
 
# Example:
kubectl delete ephemeralrunner mktech224-github-actions-runner-generic-wkdwh-runner-dt9v4 -n gh-actions-runners-set --force

Step 2: Diagnose DNS Configuration

# Check CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml
 
# Test DNS resolution manually
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup api.github.com

Step 3: Fix DNS Issues

Option A: Update CoreDNS to Use Public DNS

# Edit CoreDNS ConfigMap
kubectl edit configmap coredns -n kube-system

Update the forward section to use reliable public DNS servers:

forward . 8.8.8.8 8.8.4.4 1.1.1.1

Option B: Check Network Infrastructure

Verify security groups allow outbound DNS (port 53)
Check VPC routing tables for external connectivity
Ensure firewall rules permit DNS traffic
Validate cloud provider DNS service status

Step 4: Restart CoreDNS

# Restart CoreDNS pods to apply configuration changes
kubectl rollout restart deployment/coredns -n kube-system
 
# Wait for rollout to complete
kubectl rollout status deployment/coredns -n kube-system

Step 5: Verify DNS Resolution

# Test DNS resolution again
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup api.github.com
 
# Should return successful resolution:
# Server:    10.96.0.10
# Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
# Name:      api.github.com
# Address 1: 140.82.114.5

Step 6: Monitor Runner Recovery

# Watch for new runners being created successfully
kubectl get ephemeralrunner -n gh-actions-runners-set -w
 
# Check controller logs for successful API communication
kubectl logs <controller-pod-name> -n <controller-namespace> -f

Verification and Testing

Test 1: DNS Resolution

# Verify external DNS resolution works
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup github.com
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup api.github.com

Test 2: Runner Creation

# Monitor new runner creation
kubectl get ephemeralrunner -n gh-actions-runners-set
 
# Runners should show "Running" status instead of "Failed"

Test 3: Controller Health

# Check controller logs for successful GitHub API calls
kubectl logs <controller-pod-name> -n <controller-namespace> --tail=20
 
# Should see successful token acquisition and runner registration

Prevention Strategies

1. DNS Monitoring

Set up monitoring for DNS resolution:

# Create a DNS monitoring pod
kubectl create deployment dns-monitor --image=busybox -- /bin/sh -c "while true; do nslookup api.github.com; sleep 60; done"

2. CoreDNS Resource Limits

Ensure CoreDNS has adequate resources:

resources:
  limits:
    memory: 256Mi
    cpu: 200m
  requests:
    memory: 128Mi
    cpu: 100m

3. Backup DNS Configuration

Always configure multiple upstream DNS servers in CoreDNS:

forward . 8.8.8.8 8.8.4.4 1.1.1.1

4. Network Policies

Ensure network policies allow DNS traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Key Takeaways

DNS failures cascade: DNS issues affect external API communication, causing runners to fail and become stuck
Controller dependency: The GitHub Actions controller requires reliable DNS to manage runner lifecycle
Manual intervention required: Failed runners must be force-deleted when DNS prevents normal cleanup
Infrastructure focus: Most DNS issues stem from network infrastructure misconfiguration
Monitoring importance: Proactive DNS monitoring prevents extended outages

This systematic approach identifies the root cause, resolves the immediate issue, and implements preventive measures to avoid future occurrences.

My Notes

Explorer

GitHub Actions Runner DNS Failures