Understand the Nature of Toil and Where It Comes From

Infrastructure teams operate under a different rhythm than product teams. The work is often invisible until something breaks, and the constant pressure of keeping systems running can crowd out the very improvements that would reduce that pressure. Many infrastructure managers find themselves caught in a cycle of putting out fires, with little time to step back and address root causes. Breaking this cycle requires a deliberate shift in how you allocate time, set priorities, and communicate value.

Toil is the manual, repetitive, and often automatable work that keeps systems running but does not produce permanent improvement. It includes pager rotations, manual data migrations, restarting services, and responding to the same alert patterns week after week. Before you can manage it, you have to see it. Ask your team to track how much of their time goes into toil versus project work. A simple week-long diary study often reveals that a large fraction of effort goes into tasks that could be automated or eliminated with a small investment. The goal is not to eliminate all toil overnight, but to understand where it lives so you can decide which pieces to address first.

Create a Time Budget for Improvement Work

Once you have a clear picture of toil, the next step is to carve out a protected time budget for reduction and improvement. This is one of the most important decisions an infrastructure manager can make. Without dedicated space, improvement work will always lose to urgent tickets. A commonly recommended starting point is 20 percent of team capacity, but the exact number matters less than the principle that this time is non-negotiable.

To sell this to product stakeholders, connect the investment directly to business outcomes. Explain that every hour spent reducing toil today translates into faster delivery, lower risk, and reduced operational cost tomorrow. Use concrete examples from your own team. For instance, if a manual deployment process takes two hours per week, automating it frees up those hours and reduces the chance of human error that could cause downtime. Show the math. When stakeholders see that improvement work pays for itself in a quarter or two, they are far more likely to support it.

Use Service Level Objectives as a Decision Tool

Service Level Objectives, Service Level Indicators, and Error Budgets are not just reliability metrics. They are prioritization tools that help you make clear trade offs between reliability and velocity. When your team has an error budget remaining, you can confidently ship new features. When the budget is depleted, you shift focus to reliability work. This creates a transparent and objective way to decide whether to defer infrastructure improvements in favor of product features or vice versa.

Implementing SLOs does not have to be a massive project. Start with one or two critical services. Define a meaningful SLI, such as request latency or error rate, and set a target. The key is to make the SLO visible to both the team and your stakeholders. Include it in your regular reviews. When stakeholders ask why the team is not delivering a new feature, you can point to the error budget and discuss the consequences of exceeding it. This removes the feeling of arbitrary decisions and replaces it with a shared understanding of risk.

Align Infrastructure Work with Business Outcomes

Infrastructure teams sometimes struggle to articulate their value in business terms. A platform upgrade or a migration to a new backend does not sound as exciting as a customer facing feature, but these projects directly affect reliability, scalability, and cost. As a manager, your job is to translate technical metrics into outcomes that non technical leaders care about.

For example, reducing page load time by 200 milliseconds might increase conversion rates. Cutting cloud costs by 15 percent directly improves the bottom line. Improving uptime from 99.9 percent to 99.99 percent reduces revenue risk and customer churn. When you frame infrastructure work in terms of revenue, cost, risk, or customer experience, you move from being a cost center to a strategic partner. Use these translations in planning meetings, budget discussions, and executive updates. Over time, stakeholders will begin to see infrastructure investments as essential enablers, not optional overhead.

Design Career Growth Paths That Match Infrastructure Work

Infrastructure engineers often face a career ladder designed for product engineers, which values visible feature delivery over behind the scenes reliability work. This mismatch can lead to demotivation and turnover. To retain skilled infrastructure engineers, you need a career progression that recognizes depth in systems thinking, automation, incident analysis, and operational excellence.

Work with your HR or engineering leadership to define levels that explicitly value these contributions. For example, a senior infrastructure engineer might be expected to design self healing systems, lead incident post mortems, or mentor others on reliability practices. Promotions should be based on demonstrated impact in these areas, not on the number of features shipped. When engineers see that their work is valued and leads to growth, they are far more invested in the long term health of the infrastructure.

Distribute Incident Ownership to Prevent Burnout

In many infrastructure teams, the most experienced engineers end up owning every incident because they know the systems best. This creates a single point of failure and burns out your top talent. Instead, distribute incident ownership across the team with a clear rotation and escalation path. Use primary and secondary on call roles so that knowledge is shared and no one person shoulders the full load.

Runbooks are essential for this. When every incident response follows a documented procedure, even less experienced engineers can handle common problems confidently. Invest time in writing and maintaining runbooks for your top incidents. Review them after each incident to capture lessons learned and update the documentation. This not only reduces the cognitive burden on the team but also makes on call more manageable for everyone.

Foster a Culture of Learning from Failures

Infrastructure teams that operate in a culture of blame learn to hide mistakes. This is dangerous because hidden failures lead to larger ones. A blameless post incident review process focuses on what the system and processes allowed to happen, not on who made a mistake. The goal is to improve the system so the same failure cannot happen again.

Conduct reviews within a few days of each significant incident. Include the full timeline, what was known at each step, what decisions were made, and why. Write up actionable recommendations with clear owners and due dates. Track those recommendations over time to ensure they are implemented. When the team sees that every incident leads to measurable improvement, they will be more willing to report issues early rather than sweeping them under the rug.

Measure What Matters for Infrastructure Teams

Without measurement, it is hard to know whether your management practices are working. Choose a small set of metrics that reflect the health of your team and your systems. The DORA metrics are a good starting point: deployment frequency, lead time for changes, mean time to recover, and change failure rate. These apply to infrastructure teams as much as to product teams, especially if you treat infrastructure changes as code.

Additionally, track toil percentage, cost efficiency, and the number of recurring incidents. Review these metrics monthly with the team and quarterly with stakeholders. Use the data to guide your decisions. If toil percentage is high, increase the improvement time budget. If change failure rate rises, slow down deployments and tighten testing. Measurement gives you a way to validate your instincts and demonstrate progress.

Managing an infrastructure team is not about doing more of the same operational work. It is about systematically reducing the work that does not need to exist, investing in automation and reliability, and communicating the value of that investment to the rest of the organization. By focusing on these areas, you can move your team from a cycle of firefighting into a sustainable rhythm of continuous improvement that benefits both the engineers and the business.