Engineering & Technology - Engineering
Software Engineer - Reliability
IC / Practitioner
Independently auditedBuilds and operates the software the business runs on. Owns a defined area of the function and the outcomes it produces.
Audited package
Built, independently audited role package. Everything on this page is written for this seat.
Your Stack (Swappable)
This role works against whatever tools you already run. We list the tool categories the role operates in, never a named vendor, so you are never asked to replace a system that works.
- Service and client frameworks
- Service mesh and proxy controls
- Fault injection and chaos
- QA & Testing
- Feature and configuration control
- Observability & Monitoring
Deliverables
- Design documents
- Review checklists
- Technical standards
- Delivery and sequencing plans
Responsibilities
- Design and build reusable deadline, retry, idempotency, circuit, bulkhead, concurrency, queue, backpressure, load-shed, and degradation mechanisms.
- Create service resilience contracts and failure-mode matrices that define user behavior, dependency assumptions, containment, recovery, and ownership.
- Build fault-injection and deterministic failure test harnesses for latency, error, partition, saturation, restart, duplicate, and recovery states.
- Integrate mechanism telemetry and state with observability standards while leaving platform and pipeline ownership to Observability.
- Measure false trips, protection, p50-p999 overhead, CPU, memory, queues, throughput, saturation, recovery, and user impact under representative workload.
- Migrate services to safer mechanisms through compatibility, canary, feature control, stop, rollback, exception tracking, and cleanup.
- Diagnose retry storms, cascading failure, queue collapse, timeout leaks, circuit oscillation, unsafe fallback, and failed recovery with call-graph evidence.
- Partner with SRE, Service, Product, Platform, Observability, Security, Quality, Incident, and Performance owners without absorbing their mandates.
The essence of this seat
The Software Engineer - Reliability is an IC individual contributor and the engineer who builds resilience into services through reusable software, libraries, runtime controls, failure contracts, fault injection, and safe degraded behavior.
Signature move
Map the user operation and dependency graph with deadlines, retries, queues, concurrency, side effects, and failure propagation before adding a resilience pattern
What good looks like
composes deadlines and retry budgets across the graph and prevents multiplicative load
Decision rights
- - - Design and implementation of approved resilience libraries, runtime mechanisms, test harnesses, safe defaults, migration tooling, and mechanism telemetry
Working vocabulary
What this role does
The Software Engineer - Reliability is an IC individual contributor and the engineer who builds resilience into services through reusable software, libraries, runtime controls, failure contracts, fault injection, and safe degraded behavior. The role's working domain is deadlines and timeout propagation, retry budgets and jitter, idempotency, circuit breakers, bulkheads, concurrency and queue limits, backpressure, load shedding, graceful degradation, dependency isolation, fault injection, recovery ramp, and reliability-mechanism adoption. The seat is accountable for services fail in bounded, observable, reversible ways without retry storms, hidden queues, cascading dependency failure, unsafe fallbacks, or untested recovery paths. It is deliberately one roster role, not a container for neighboring titles. The boundary is behavioral and operational: this seat builds resilience into service code, shared libraries, runtime mechanisms, and tested failure behavior; the Site Reliability Engineer owns SLOs, error budgets, on-call, incident practice, and the broader reliability operating discipline; this seat builds mechanisms that contain, degrade, retry, backpressure, and recover from failure; the Software Engineer - Observability builds the metrics, logs, traces, telemetry pipelines, and debugging platform used to observe behavior; this seat builds reusable resilience patterns and cross-service failure controls; the Backend Engineer owns application service business logic, APIs, data behavior, and product-specific implementation. What separates a strong Software Engineer - Reliability from a generic engineer is domain judgment under failure. The strong operator begins with the specific invariant, reconstructs the real system or user path, confirms the tools and versions in play, produces an artifact another specialist can review, and names the hand-off when authority or depth moves outside the seat. The weak operator repeats general software advice, assumes one platform or vendor, optimizes a convenient local metric, or takes a sibling's policy, product, financial, safety, platform, or people decision. The core emphasis for this package is service-level resilience mechanisms, end-to-end deadline and retry composition, proven idempotency, circuit breakers and bulkheads, bounded queues and backpressure, load shedding and approved degradation, dependency containment, safe fault injection, controlled recovery ramps, and the build-mechanism-versus-operate-reliability boundary.
Best-fit customer
engineering teams that want consistent judgement applied to design, review, and delivery
Need more than one role?
Buying the whole team or the complete department lowers your effective cost per role at every tier of the ladder.