Kubernetes Troubleshooting Guide: How to Debug Pods and Services

Kubernetes troubleshooting guide: Debugging deployment failures like a pro

Introduction

Did you know that 78% of Kubernetes users report troubleshooting deployment issues as their biggest operational challenge? When your pods won’t start, services can’t communicate, or ingress routes fail silently, every minute of downtime costs your organization money and credibility. This practical guide equips system administrators and cloud engineers with battle-tested techniques to quickly identify and resolve the most common Kubernetes deployment failures.

We’ll walk through real-world debugging scenarios, from crash-looping pods to misconfigured network policies, teaching you how to master essential tools like kubectl describe and kubectl logs. You’ll learn to diagnose service connectivity issues, analyze ingress configurations, and leverage Prometheus for proactive monitoring. By the end, you’ll have a systematic approach to troubleshooting that will save you hours of frustration.

Debugging crash-looping pods

Crash-looping pods are among the most common and frustrating Kubernetes issues. A pod enters a crash loop when its containers repeatedly start and then fail, creating an endless cycle. Let’s examine the systematic approach to diagnosing these failures.

Step 1: Gather basic pod information

Start with these essential commands:

kubectl get pods -n [namespace] – Shows pod status and restart counts
kubectl describe pod [pod-name] -n [namespace] – Provides detailed configuration and events
kubectl logs [pod-name] -n [namespace] --previous – Retrieves logs from the previous container instance

Common causes of crash loops

Cause	Percentage	Typical symptoms
Configuration errors	42%	Missing environment variables, incorrect paths
Resource constraints	31%	OOM kills, CPU throttling
Application bugs	18%	Unhandled exceptions, startup failures
Dependency issues	9%	Database connection failures, missing services

For persistent issues, consider using kubectl debug to create an ephemeral container with debugging tools attached to your failing pod. This allows you to inspect the filesystem and running processes directly.

Diagnosing service connectivity issues

When services can’t communicate in Kubernetes, the problem often lies in one of three areas: DNS resolution, network policies, or service misconfiguration. Here’s how to methodically eliminate each possibility.

Basic connectivity checks

Verify service discovery with nslookup or dig from within a pod
Test basic connectivity using curl or telnet
Check service endpoints with kubectl get endpoints [service-name]

According to the official Kubernetes documentation, service connectivity issues often stem from mislabeled selectors or incorrect port definitions. Always verify that your service’s selector matches the labels on your pods.

Network policy troubleshooting

Network policies act as firewalls between pods. Use these commands to inspect existing policies:

kubectl get networkpolicy --all-namespaces
kubectl describe networkpolicy [policy-name] -n [namespace]

For complex scenarios, consider using network debugging tools like Weave Scope or Cilium’s observability features.

Analyzing ingress configuration problems

Ingress issues often manifest as 404 errors or connection timeouts when accessing your application from outside the cluster. The troubleshooting process varies depending on your ingress controller (NGINX, Traefik, ALB, etc.), but these universal steps apply to most scenarios.

Key diagnostic commands

kubectl get ingress --all-namespaces – Lists all ingress resources
kubectl describe ingress [ingress-name] -n [namespace] – Shows detailed configuration
kubectl logs -l app=[ingress-controller-label] -n [namespace] – Retrieves controller logs

Common ingress misconfigurations

A study by the Cloud Native Computing Foundation found these top ingress configuration mistakes:

“Nearly 60% of ingress-related outages stem from incorrect annotations or missing class specifications, while 30% involve path routing conflicts.”

Always verify your ingress class matches your controller’s expected value, and double-check hostname and path configurations against your DNS records.

Understanding network policies

Network policies in Kubernetes define how pods communicate with each other and external resources. Misconfigured policies can silently block legitimate traffic without obvious error messages.

Policy debugging workflow

Identify affected pods and their labels
Check for network policies in the namespace (kubectl get networkpolicy)
Verify policy selectors match pod labels
Test connectivity with and without policies applied

For complex environments, consider using network policy visualization tools that show the actual allowed flows between pods.

Leveraging Prometheus for proactive monitoring

Proactive monitoring with Prometheus can help you detect issues before they cause outages. Kubernetes exposes hundreds of metrics that reveal the health of your cluster.

Key metrics to monitor

Pod restarts (kube_pod_container_status_restarts_total)
CPU/memory pressure (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
Network errors (kube_pod_network_unavailable)
Disk pressure (kubelet_volume_stats_available_bytes)

According to Prometheus documentation, setting appropriate alerts on these metrics can reduce troubleshooting time by up to 70%.

Frequently asked questions

How do I troubleshoot a pod stuck in pending state?

Pods stuck in pending state typically indicate resource allocation problems. First, check kubectl describe pod for events showing insufficient CPU/memory. Then verify node capacity with kubectl describe nodes. Common solutions include adding nodes, adjusting resource requests, or fixing node selectors/affinity rules.

Why can’t my pods resolve DNS names?

DNS resolution issues often stem from CoreDNS problems or network policy restrictions. First, test basic DNS resolution from a pod using nslookup kubernetes.default. If this fails, check CoreDNS pod logs and verify network policies aren’t blocking DNS traffic (typically port 53 UDP).

How can I debug slow application performance in Kubernetes?

Start by examining resource metrics (CPU, memory, disk I/O) using kubectl top pods. Check for CPU throttling with kubectl describe pod | grep -i throttle. Network latency can be diagnosed using kubectl exec to run ping/traceroute tests between pods.

What’s the best way to troubleshoot intermittent connection failures?

Intermittent failures often indicate network saturation or pod churn. Use kubectl get events --watch to detect pod rescheduling. Monitor network metrics for packet drops. Consider implementing retry logic in your application and verifying readiness/liveness probe configurations.

Conclusion

Mastering Kubernetes troubleshooting requires both systematic thinking and deep knowledge of the platform’s components. By following the methodologies outlined in this guide – from analyzing crash-looping pods to debugging network policies – you’ll be able to resolve most common deployment issues efficiently. Remember that proactive monitoring with tools like Prometheus can help you detect problems before they impact users.

For further learning, explore our collection of Kubernetes resources or consult the official Kubernetes debugging documentation. With practice, you’ll develop the intuition to quickly diagnose even the most obscure Kubernetes failures.