Senior Software Engineer, Observability

About the Role: 

We are looking for a Senior Software Engineer to join our Observability team and help build the platform that gives Redpanda’s engineering organization deep visibility into the health, performance, and behavior of our systems. You will own and evolve our Grafana-based observability stack—spanning metrics, logs, and traces—and ensure that every team at Redpanda has the tooling and insights they need to ship reliable, high-performance software.

This is a high-impact role at the intersection of infrastructure and developer experience. You will work closely with platform and product engineering teams to design scalable observability solutions, drive adoption of best practices, and reduce mean time to detection and resolution across our cloud and on-premise deployments.

You Will:  

• Design, build, and maintain Redpanda’s observability platform using the Grafana stack (Grafana, Mimir, Loki, Tempo, Alloy/Agent)

• Develop and optimize dashboards, alerts, and SLO/SLI frameworks that give engineering teams actionable insights into system health

• Build and operate scalable metrics, logging, and distributed tracing pipelines that handle high-cardinality data across cloud and on-premise environments

• Instrument services and infrastructure with OpenTelemetry to ensure comprehensive, standards-based telemetry collection

• Partner with platform teams to improve incident detection, root-cause analysis, and mean time to resolution (MTTR)

• Evaluate and integrate new observability tools and techniques, driving continuous improvement of our monitoring capabilities

• Contribute to internal tooling and automation that streamlines observability onboarding for engineering teams

• Participate in on-call rotation to keep observability infrastructure running and incident free

You Have: 

• 5+ years of experience in software engineering with a focus on observability, monitoring, or infrastructure

• Deep hands-on experience with the Grafana stack (Grafana, Mimir/Prometheus, Loki, Tempo) in production environments

• Strong understanding of metrics, logging, and distributed tracing paradigms and their trade-offs at scale

• Experience with OpenTelemetry (OTel) for instrumentation and telemetry collection

• Proficiency in Go and Python

• Experience running and operating infrastructure on Kubernetes in public cloud environments (AWS, GCP, or Azure)

• Comfortable working with a 100% distributed engineering team, collaborating on GitHub, etc.

• Experience with AI coding tools (e.g., Claude Code) and able to independently validate, refine, and productionize generated outputs

• Solid understanding of time-series databases, log aggregation systems, and query languages (PromQL, LogQL)

Nice to Have: 

• Strong understanding of Go

• Experience operating a SaaS platform with production observability at scale

• Familiarity with eBPF-based observability or continuous profiling tools (e.g., Pyroscope, Parca)

• Experience with infrastructure-as-code (Terraform, Pulumi) and GitOps workflows

• Operated and used streaming platforms (e.g., Kafka, Redpanda) either as a user or provider

• Experience building or managing multi-tenant observability platforms

• Contributions to open-source observability projects (Grafana, Prometheus, OpenTelemetry, etc.)