Staff Software Engineer , Observability

The company is a community of communities. It’s built on shared interests, passion, and trust, and is home to the most open and authentic conversations on the internet. Every day, the company users submit, vote, and comment on the topics they care most about. With 100,000+ active communities and approximately 130 million daily active unique visitors, the company is one of the internet’s largest sources of information. For more information, visit [link for members] .

This role is fully remote. The company has a flexible first workforce.

The Observability (OBS) team is looking to hire an Engineer that thrives at the intersection of infrastructure and software development. This team owns a suite of tools for allowing engineers to understand their creations, based primarily on open-source solutions at scale. We’re active users of and contributors to our open source tools, including Prometheus, Thanos, and more.

Monitoring

Our monitoring stack at the company processes billions of data-points a minute, delivering insights at scale. The company has one of the larger deployments of Prometheus and Thanos in the world , and with this come unique challenges of scale for these systems. Fun problems include performance engineering on a distributed query system and product thinking around new features to remove the user pain from this stack.

Logging

We operate a platform for logging based on Vector and Clickhouse that processes millions of logs a day for the company. Thousands of people at the company use this system every day as they maintain, deploy, and debug their code. Join us as we invest in this system to help users better understand the data they have.

Distributed Tracing

Tracing at the company is based on OTEL, Clickhouse, and Grafana. It’s light on features, but currently processing tens of millions of events, and we will be building new ways for users to leverage and understand this data in the coming year.

As a member of the Observability team, your work will span these domains, which are rich with challenging infrastructure and software engineering problems. Your work will directly impact hundreds of millions of users around the world. Join us and help build the future of the company!

In your day-to-day, you can expect to

• Work collaboratively with a team of software engineers to create and maintain the foundational platform for running the company’s infrastructure.

• Deliver software to improve the availability, scalability, latency, and efficiency of observability components.

• Contribute feedback to the technical and strategic direction of eventing at the company.

• Automate critical aspects of the event driven development process

• Share on-call responsibilities.

• Contribute upstream changes to the open source projects we use

You have

• 7+ years of experience developing internet-scale software, preferably in the context of infrastructure.

• Familiarity with distributed systems development, bonus if familiar with any of the specific tools (Prometheus, Thanos, Grafana, Vector, Clickhouse, Otel, Loki)

• Experience developing on top of Kubernetes or similar distributed systems.

• Kubernetes controller or operator development experience is a huge plus.

• Strong troubleshooting capabilities surrounding both systems and software.

• Experience engineering large systems, tracking work, and being a self-starter on projects.

• Excellent communication skills to collaborate with a service-oriented team and company.

Benefits

• Comprehensive Healthcare Benefits and Income Replacement Programs

• 401k with Employer Match

• Global Benefit programs that fit your lifestyle, from workspace to professional development to caregiving support

• Family Planning Support

• Gender-Affirming Care

• Mental Health & Coaching Benefits

• Flexible Vacation & Paid Volunteer Time Off

• Generous Paid Parental Leave

Pay Transparency

This job posting may span more than one career level.

In addition to base salary, this job is eligible to receive equity in the form of restricted stock units, and depending on the position offered, it may also be eligible to receive a commission. Additionally, the company offers a wide range of benefits to U.S.-based employees, including medical, dental, and vision insurance, 401(k) program with employer match, generous time off for vacation, and parental leave. To learn more, please visit [link for members] .

To provide greater transparency to candidates, we share base salary ranges for all US-based job postings regardless of state. We set standard base pay ranges for all roles based on function, level, and country location, benchmarked against similar stage growth companies. Final offer amounts are determined by multiple factors including, skills, depth of work experience and relevant licenses/credentials, and may vary from the amounts listed below.

The base salary range for this position is

$217,000 — $303,900 USD

In select roles and locations, the interviews will be recorded, transcribed and summarized by artificial intelligence (AI). You will have the opportunity to opt out of recording, transcription and summarization prior to any scheduled interviews.

During the interview, we will collect the following categories of personal information: Identifiers, Professional and Employment-Related Information, Sensory Information (audio/video recording), and any other categories of personal information you choose to share with us. We will use this information to evaluate your application for employment or an independent contractor role, as applicable.  We will not sell your personal information or disclose it to any third party for their marketing purposes.  We will delete any recording of your interview promptly after making a hiring decision.  For more information about how we will handle your personal information, including our retention of it, please refer to our Candidate Privacy Policy for Potential Employees and Contractors .

The company is proud to be an equal opportunity employer, and is committed to building a workforce representative of the diverse communities we serve.  The company is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If, due to a disability, you need an accommodation during the interview process, please let your recruiter know.