Fulfilling the Promise of Automation
The urge to shrink the gap between wanting and having is deeply human.
Genies, magic wands, dreams – all are versions of the same wish: let it simply be so.
Automation is that wish applied to work: if only completing tasks could be as easy as wishing them done – no risk, effort, or grind.
It’s even a fixture in our sci-fi: in Star Trek, the commander simply says “Computer …” and it’s done. Machines replicate objects, heal ailments, navigate space, and even govern, with humans simply acting as beneficiaries.
We’re still a long way from that vision. While automation has made huge strides over the last two centuries, implementing it remains challenging, risky, expensive, and unreliable.
I’ve been passionate about automation since my first engineering job back in 2008 as an Automation Engineer – there’s something profoundly satisfying about automating away unnecessary manual toil.
At its core, automation is about reclaiming wasted time – the scarcest resource in each of our lives.
In this in-depth guide, I’ll share what I’ve learned about executing automation projects successfully: both the strategic principles and the technical practices that make them work.
The dream of automation versus reality
If automation holds so much promise, surely companies are automating everything they can and reaping the rewards – right? Not even close.
Companies typically show limited interest in automation. Many are aware, often from painful experience, that executing automation projects successfully is challenging, expensive, and risky.
The typical company will only contemplate automation when it seems almost risk-free, plug-and-play, and low cost. (I believe this is the main reason why chat-bot-style LLM tools have seen such rapid corporate uptake – more so than performance considerations or efficiency pressures from tighter monetary policy; They’re a product that’s quick, easy, comfortably priced, and safe to deploy.)
Yet this stance leaves competitiveness on the table. Over a long enough time horizon, companies with automated systems will steadily displace their less-automated rivals.
To achieve significant efficiency gains, we first need to understand the obstacles that make automation so challenging to implement – and what can go wrong when it’s done poorly.
Time preferences
Automation initiatives carry an upfront cost and ongoing maintenance burden, with an uncertain future gain.
Any resources a manager diverts to an automation project must come from other priorities – the projects other stakeholders are demanding updates on, and which will feature in this year's performance review.
A successfully shipped feature that earns a positive reception, or a bug fix that satisfies an important or frustrated client, brings immediate and visible recognition to those involved.
A successful automation project, however, delivers gains that are delayed, diffuse, and indirect. People quickly forget the time it now saves or the performance gains it now brings. They also don't appreciate the interruptions it causes them when introduced, nor the learning and changes in routine it requires.
Furthermore, in software engineering, automation is often a delayed payoff on top of a delayed payoff. It’s therefore unsurprising that automation on the GTM side of the business – i.e. in RevOps – receives more support.
As a result, delivering on urgent and highly visible projects is a surer path to recognition and rewards than quietly increasing the average pace of delivery behind the scenes – even if the latter ultimately produces a larger impact over the life of the company.
This can create a dynamic whereby teams appear to be delivering effectively, with a steady stream of shipped feature celebrations, while the company slowly loses ground to competitors (which includes customers developing their own solutions instead), and increments of growth get harder and harder to achieve.
At the heart of this issue is that decisions about time preferences sit with company leadership, whereas awareness of potentially valuable automation initiatives lies with those at the bottom of the org chart – where repetition and friction are felt during day-to-day work.
Without a mechanism from the top that empowers those below to automate, most efforts will be quietly stymied by the relentless force of “this is more urgent”.
Apprehension of change
Those who choose full-time employment generally do so because they value stability and structure – not risk and uncertainty. This makes automation feel more like a portent than a blessing.
Some fear automation will cost them – or others – their jobs. But for many, automation simply represents a disruption to routine, and they'd find it more comfortable if it didn’t happen at all.
This creates a baseline resistance to automation, a steady headwind against such initiatives.
That resistance is not mild. I’ve seen multiple cases throughout my career where teams continued performing tedious, repetitive tasks manually long after an automated tool was built for that exact purpose. Often they didn’t even try the tool, despite multiple public reminders of its existence and assurances that technical support would be available at all times.
And this was in software engineering – where you'd expect the most enthusiasm for automation. Programming is, after all, the essence of automation: instructing machines to do the work for us.
Similar patterns have been seen in the adoption of AI tools, sometimes resulting in explosive reactions.
There are several understandable reasons behind this apprehension. When people haven’t built a tool themselves, it’s not a familiar part of their workflow. It’s a foreign object. They’ll have to learn how to use it, and if something goes wrong, they’ll be stranded with a situation they don’t know how to fix.
Guarantees of support rarely change this dynamic. In fact, they reinforce a sense of disempowerment – “before, I could do this myself; now I’m a trainee”.
Automation projects that ignore these human factors are likely to fail, deepening leadership’s distrust of automation efforts.
Failure risk
A manager who signs off on an automation project exposes themselves to several risks. It could fail outright; it could run over its estimates or budget and look like a waste of resources; or worse, it could cause operational problems during launch and trigger complaints.
These risks are real. They may have been burned before, or seen it happen to a colleague. It feels safer to simply maintain the status quo.
Personal disincentives
Sharing automation solutions is often disincentivized. If an employee works out how to automate half of their day – why would they share it?
By keeping it to themselves, they reap the benefits personally. The internet is full of such “I automated my job and now play Solitaire all day” stories.
But if they share it, they just get a rise in expectations instead, nullifying its personal benefits. In some cases they may even be reprimanded for going off task (raises hand – ironically when working at an industrial automation company).
Further, due to the frequent perception of automation as a source of redundancy, they may even fear sharing it.
Finally, there’s the problem of attribution. If an automation project subsequently increases the pace of project delivery, the recognition goes first to those who ship the projects. The automator ends up at the bottom of the credits – if they’re lucky.
Not Invented Here syndrome
New tools are usually developed by one team or a single individual, with the hope that other teams will adopt them. The second step is almost always harder than the first.
This matters because automation in most companies is done informally and locally (and often secretly).
Applying the Pareto principle, you’d expect that without managerial pressure, around 20% of people will try the tool and 80% will not. That matches my experience.
This means that even if a tool successfully automates a task, by default you’ll only capture about 20% of its value – which makes its development uneconomical.
Automating successfully
So how can a company successfully start and sustain an automation program given all of these obstacles? By enacting the following core practices:
1. Supportive strategy. Automation must be part of the company’s broader business strategy – enabled, funded, and reinforced from the top.
2. Careful selectiveness. Because automation projects carry risk, they should be chosen carefully and held to a high bar for impact and feasibility.
3. Strong project management. Automation initiatives require disciplined Agile project management to ensure they get delivered effectively. Failed projects quickly erode trust, which can stall the entire program.
4. Reliable technical design. Even minor unreliability will amplify resistance to use. The tools must be robust enough to leave no excuse for not using them.
5. Steadfast change management. There must be an unrelenting push to bring automation solutions into widespread use. The completion of a solution doesn’t mark the end of the journey, but the beginning of the final mile.
Introduction of an automation strategy
To counteract short-termism, automation must be explicitly supported by company leadership. The engineering strategy (and indeed the strategy for any other department) should be designed with a long-term upward trend in delivery pace in mind.
This is not common. Most strategies focus on first-order goals – i.e. engineering will achieve X. Few address second-order goals – i.e. how engineering will achieve an increasing amount of X per unit cost over time, or how engineering will maximize outcomes like X over the next five years.
There exist exceptions, like the LinkedIn CTO who courageously halted new development for two months – a bold move that cost little compared to the time saved thereafter. But such exceptions prove the rule.
There will always be strong voices prioritizing the near term before all else. They should be reminded that in accretive work – as most software development is – what determines how much we ship this month isn’t how hard we push right now, how narrowly we focus right now, or how much low-urgency work we cut right now.
No, it’s the accumulated slowdown created by every time in the past we took that approach.
Only through a formal strategy – and policies enacting that strategy – can this tendency be counteracted.
A simple automation strategy is a general efficiency improvement strategy with:
- a defined hurdle payback period;
- a minimum and maximum quota band; and
- a requirement that initiatives are ordered by fastest payback.
As an example: “The engineering department as a whole is to spend between 10% and 20% of its resources per quarter improving its pace of development. All related initiatives must estimate their payback period, be tracked in the same manner as other projects, and review the accuracy of their estimates after completion. No initiative with a payback period longer than 12 months may proceed.”
The mandatory quota band is especially important – it’s what protects these initiatives from being perpetually deprioritized.
How the strategy is implemented can be left up to teams. But as they pursue fast payback periods, they will quickly realize that automation projects only meet the hurdle period if they’re broadly useful, widely adopted, and relatively quick to build.
Selection of automation initiatives
The best automation project is the one that never happened: it wasn’t worth the effort, a simpler solution was found, or the underlying task no longer needs doing.
You automate not when it’s a bit better than manual, but when it's vastly better.
The hurdle for an automation project should be in the region of 10x – saving 90%+ of time spent on a task. This creates a margin of safety. (That threshold is not difficult to reach; most copy-paste routines consume far more than 10x the time an automated system would.)
Automation is also a late step; process design and simplification must come first.
Process simplification is a neglected practice. Processes tend to grow in complexity because it’s faster to add to them than to reform them; reform requires deeper comprehension.
Yet it’s essential to simplify long before automating, because doing so makes the automation far cheaper, and simplifying or adjusting complex automated processes later is difficult and time-consuming (i.e. automating bad processes just cements bad processes at great expense).
Process simplification should be approached like taking a mower to an overgrown backyard: cut ruthlessly. Delete anything that can go. Once you think you’ve got it as simple as possible, go over it once more.
Simplification is hard, but if you persist, you can usually whittle a process down to a small fraction of its original size. And if that feels risky, remember: you can always add things back later.
A major obstacle to simplification is authority. An engineer can unilaterally develop automation, but rarely has unilateral authority over processes. And to an engineer, the peoplecraft required to bring about process change is harder and less enjoyable than simply automating the existing overly-complicated processes.
As a result, they tend to automate processes instead of fixing them, creating automation that further entrenches the very inefficiencies it should remove.
In fact, people are reluctant to edit existing process documents at all. It feels confrontational – like defacing someone else’s work. And deleting content – the most valuable act of simplification of all – feels especially antagonistic.
The solution is leadership sponsorship. Leaders should make it explicit that existing processes should be simplified as much as possible, and then delegate that responsibility downwards. Working agreements can be an effective tool for facilitating this.
Original authors should also signal their support for rework openly – a simple “please have at it” removes the awkwardness and frees others to make improvements guilt-free.
The project management of automation
In the absence of a defined automation program or dedicated automation teams, most automation is developed informally, privately, and often secretly. Every organization has at least one spreadsheet that has grown into a contraption.
Most automation also remains highly localized. Even if it’s offered to other teams, the “not invented here” effect means they seldom take it up. This means only a small fraction of its potential value gets captured.
Even when a dedicated team exists – such as a Platform or DevEx team – automation projects can slip outside the company’s usual Agile guardrails, since they aren’t customer-facing.
This can lead to over-sized “big bang” projects that drag on too long, miss other teams’ real needs, and sometimes serve as tech playgrounds.
To steer automation projects around these pitfalls, the following guardrails should be applied.
All automation stays on the books. Automation work – like any other efficiency improvement – should follow the same work-management practices as everything else: tickets on the board, estimation, planning, reviews, and so on.
This is a trust exercise more than anything else, and it only works if supported from above by a clear automation strategy. Otherwise, it only takes a few “that’s not a priority right now” for all the automation work to disappear back into the bushes.
With a clear strategy and visible support from leadership, the company can restore trust, transparency, and alignment.
This leads us to the next guardrail.
Deprioritisation shield. Like other efficiency projects, automation projects must be defended from constant deprioritization driven by the natural tendency towards short-term gains – such as shipping one extra feature or expediting a bug fix to please a key client.
The only effective way to do this is an allocation quota: a separate resource bucket reserved for this work.
Short-termism is a powerful force – it will push efficiency improvement tickets aside in favor of other work at every opportunity. An organization can talk endlessly about wanting efficiency and high performance, but at the end of the day, the familiar chorus of “that’s not as urgent” ensures those tickets end up at the bottom of the backlog forever.
This deprioritization effect can only be countered by a greater counter-force.
There are urgent tasks now, and there will be urgent tasks in the future. But what most affects a company’s ability to complete those urgent tasks quickly isn’t how determined or focused people are right now – it’s the investments made leading up to now. Pulling resources from the future to the present at every opportunity is like trying to win a marathon by sprinting the entire way.
Since time preference responsibility lies at the top of the organization, that is where automation quotas must be enforced from.
Quotas and their averaging windows should be written down, tracked, and reviewed using ordinary management control systems. Anyone requesting an exception should be required to record it in writing, along with their rationale.
Piece by piece. The most exciting project is a big grandiose one that will “automate all the things” – with the expectation of commensurate recognition at the end.
Instead, it needs to be thoroughly sliced into stages, starting with the fastest and highest-impact first. (Fastest should take priority – it builds faith in the project and allows the change management process to begin as early as possible.)
Still, the desire for recognition needs to be respected. Drip-fed improvements rarely receive the same treatment as something with a “launch”, so this behavior is a defensive reaction to that.
The solution is to make that launch synthetic – grouping the series of improvements together into a defined “epic”. There should be an understanding that that epic will not be suspended or expanded midstream, and once its last improvement ships, that whole epic should be celebrated.
To drive home the parity of value, frame the achievement in terms of a recent feature reference – “These ongoing savings will allow us to ship one extra Project X per year”.
Product management. Automation projects need it just as much as customer-facing functionality, and this starts with quantifying the project’s value.
Automation isn’t cheap. It carries both development and maintenance costs. So it shouldn’t be approved just because it solves one developer’s pet peeve – that botherance needs to be quantified, then compared against other efficiency improvement opportunities.
Interestingly, automation projects are often easier to value than features, because they all boil down to the same value proposition: saving resources and improving speed in the long run.
If you want to get formal, you can do IRR (internal rate of return) calculations. But a simpler measure is payback period and recurring savings. E.g.: “This project will save 100 developer hours per month, and will pay for itself in 6 months”.
Yes, developers don’t want to do this accounting-style estimation work. But it's the only way to defend automation projects effectively – especially larger ones – until trust is built up. The developers will, however, need training and support to learn this new skill.
Consolidation. Some problems repeat across teams, leading to a proliferation of solutions. Therefore, a mechanism for consolidating automation initiatives is needed.
Since bureaucracy destroys innovation, this can’t look anything like a change control board. Instead, teams should simply declare their solutions, and if they have wider application, they should gain management at the appropriate level.
If multiple existing competing solutions are discovered, hard decisions are required – some will need to be shut down or absorbed into others. Where teams have different needs, a common core can serve as a shared base from which tailored solutions extend.
I.e. non-obstructive management – no permissions involved, but reporting must be present to prevent conflicts and the waste of duplicated efforts.
Motivated participants. Automation projects should be led by people who are passionate about them.
The Pareto principle applies here – you can expect 20% or fewer to have a particular interest in automation. So find them, and offer them the chance to take on this work. They’ll also need support and supervision from senior management, due to the cross-cutting application of what they’re building.
As long as these guardrails are in place, your automation project should yield a substantial payoff.
Producing high-quality automation
Most internally developed automation I’ve seen in my career has been unreliable. It works in a narrow scenario, and fails at the slightest variance in conditions. I.e. it’s brittle.
It also fails in a way that leaves users clueless about what has happened. Worse, it leaves things in an unknown state – one the user must now decipher and clean up. Having been burned like this before, users become understandably wary of trying new tools.
To counter this tendency, every automation solution should be treated as a high-quality, professional product, no matter how small it is. It should be sturdy, accomplished by fulfilling the following qualities:
Reliability. A tool should work every time it’s invoked, regardless of who invokes it or when, and regardless of the conditions it is invoked in.
This requires carefully considering use cases and edge cases, which can be time consuming, but the alternative is having to fix them anyway after frustrating a user. Think: 1-in-100 failure rate or better. (From experience, it’s usually 1-in-10 or worse).
To help with this, automation needs automated tests, just like other software. Those tests should run regularly (e.g. through your CI system) to catch regressions early.
Simplicity. Solutions must be as simple as possible, both in technical design and user experience. Simple tools are more reliable, easier to use, and have less development risk.
Despite this, internal projects – being fresh builds (a usually rare and enticing opportunity for developers) – are often treated as an opportunity to frolic.
The latest-and-greatest frameworks, exotic languages, and clever combinations of cloud infrastructure will be eagerly suggested – along with complex design patterns, the harder to understand the better. I’ve seen a company blog end up with an architecture that looked ready to launch rockets. This must be restrained.
In fact, writing code should be the last resort. The first should be trying to accomplish the goal by configuring existing systems and tools.
If that proves impossible or insufficient, the next step is researching off-the-shelf products and services that could fit the bill.
Many organizations with engineering teams ignore the opportunity cost of developing in-house. Reinventing the wheel is common – engineering is distinctly more enjoyable for engineers than going through procurement.
If no 3rd-party offerings quite fit, the next step is to use no-code or low-code tools. Platforms like Make.com come with extensive pre-built integrations for other tools and services, and come with many of the other qualities mentioned in this section – right out of the box.
Even Google Sheets – sometimes coupled with other automation tools like Make.com – can serve as a perfectly good low-code platform in certain cases.
If it comes down to code, have it developed in the languages and frameworks already in use. If those aren’t appropriate, use the language most commonly used for that kind of work – something old enough that everyone’s heard of it and the kinks are long ironed out, not something shiny and brand new. Reliability comes first.
Finally, before any development begins, the design should be reviewed by a simplicity-favoring senior engineer, someone keen to answer the question: “is this the simplest design that could accomplish our goal?”
(I’ll add a brief note on the direct use of LLMs here. No – you can’t rely on a stochastic system by itself for most automation, because of the reliability requirements. Your automation can’t “usually mostly work”; it must work. It’s best to concretize the requirements into deterministic systems wherever possible – which are also far cheaper to run. But do use as much LLM assistance as you like when developing those deterministic systems.)
Resilience. Automation isn’t as resilient as manual work. People adapt when things don’t go to plan, and when they’re doing something manually they have immediate context if anything goes wrong.
The problem with automation is that when it breaks, it interferes with that adaptability. It leaves the user in the dark, fumbling around for the light switch. Therefore, its reliability must be in excess of usual.
It should also be fail friendly. Rather than just erroring and exiting, it should pause and tell the user which step it’s stuck on, and what’s needed to continue, so the user can cure that excursion.
If the tool does need to error and exit, it should either roll everything back to its original state, or give the user all information about the current state and tell them what to do next.
Pre-checks are a key part of this. The tool should check that all of its expectations are met before trying to do anything, so that the user can receive a much less nerve-wracking pre-execution error, rather than a painful mid-execution error that leaves things in an unknown state.
Observability. A tool should be verbose. It should log the details of every step it executes. If it fails, it should give detailed error messages, describe the current state, and guide the user as to what to do next.
In addition to being visible to the user, these logs and errors, as well as usage and failure metrics, should be saved in a central location. This supports reporting and accountability.
The change management of automation
Introducing automation – no matter how small – is a change that requires a concerted and methodical effort. It’s a project in its own right. The rollout must be planned, tracked, and backed by management’s unwavering support.
The desired change flows naturally from the value proposition defined in previous stages – we want successful adoption of the automation solution in order to realize that value.
Achieving this change requires following standard change management practices while factoring in the nuances of automation-based changes.
In the following sections I’ll use Prosci’s ADKAR model as the change management framework.
A – Awareness
The process starts by bringing everyone’s attention to the need for change. The shortcomings of the status quo, and the losses and risks from failing to change, must be communicated clearly and convincingly.
Management should be visibly present to signal their backing, and several supporters from across the organization should voice their support during the presentation, to build recognition of a broad-based desire for change.
Ample opportunities for dialogue should be provided. Everyone should be encouraged to voice their concerns and objections. This not only reveals where resistance may arise, but also generates feedback that can be used to adjust plans.
This helps blunt resistance grounded in feelings that “this was sprung on us”, or that the change is an unnecessary imposition.
Finally, respect must be paid to the solutions that have been in use up until this point. Omitting this step is a common source of avoidable resistance.
Those systems were likely an improvement over what came before. Even if present systems are clearly no longer fit for purpose, it’s important to recognize that people may be fond of them and put a lot of work into them, which contributes to their reluctance to move on. A tone of gratitude goes a long way.
D – Desire
Teams are hesitant to pick up automation developed elsewhere for several reasons: It’s unfamiliar (better the devil you know), it takes effort to learn, and it creates a risk of “what if it goes wrong”. These concerns should be met with empathy, and it should be made clear that each has been accounted for.
Everyone should get an opportunity to share their needs and preferences for the tool before it’s built. This gives them a sense of participation – they’ve played a role in shaping it, or at least tacitly accepted its requirements as-is.
To ease apprehension about using the tool, it should come with a guarantee that obligates its developers to come to the rescue if it doesn’t work. But this comes with a counter-obligation for users: to report the issue and wait, rather than immediately falling back to manual operation – to give the developers an opportunity to fix it.
Management should also address the fear of automation-caused redundancy. This concern is real for some staff and shouldn’t be glossed over.
The core of the fear is: “what if it makes us run out of work”. But most software engineering – at least in product-focused organizations – sits in R&D, where there’s always more work to do. Showing the depth of current backlogs and roadmaps, and explaining that “we’ve more work than we could ever possibly do; we need automation to speed up the pace”, goes a long way towards dispelling those fears.
In general, R&D staff should fear falling behind the competition more – something automation actually defends against. The main constraint on available R&D work isn’t automation but the cost of capital (rising interest rates discount the value of future cash flows).
K – Knowledge
Having teams use a new tool creates an added responsibility for them, and matching support should be provided to help them get up to speed.
It makes an enormous difference to have someone on a Zoom call guide them through their first uses. As they chat with them about the tool – its history, its roadmap, and so on – it helps them feel part of its story, rather than facing a foreign object plopped in front of them.
There should also be ample documentation and onboarding materials for those who prefer to self-serve. But be wary about using video guides – they’re much harder to keep up to date than text-based ones, and can quickly become misleading if the tool changes.
Given this support burden, it’s also worth considering whether it’d work better to assign the now-automated task to a specific team rather than leaving it as a shared responsibility.
A – Ability
When you replace existing routines with a tool, people’s confidence drops from a hundred to zero. They knew the manual way like the back of their hand, but they’ve no idea how the new tool works.
To help with this, the tool’s inner workings should be transparent. It should also be designed for ease of understanding and developing, using a common and familiar language, to encourage others to participate in its maintenance and development. This reinforces the idea of the tool as an extension of their skills.
Another useful approach is a “champion” system. Each team can have a champion for each automation tool, who acts as its internal expert, advocate, and connection to its main developing team.
R – Reinforcement
Once the initial trials and feedback are complete, it must be made clear that the automation will not be optional. Everyone is required to at least attempt to use it wherever it applies. And if the automation fails, users must report it.
If there is no accountability for this, the project has little chance of success.
“Please use it”, even from a senior manager, is not effective. You’ll get maybe 20% uptake – making the initiative uneconomical.
This doesn’t mean mandating an abrupt uptake – which would cause problems of its own – but steadily working towards full adoption by the target user base.
To support this accountability, the automated method should leave distinct markers compared to the manual method, and there should be periodic usage reviews, feedback gathering, and follow-ups.
But most importantly, there should be positive reinforcement. Seek out users who’ve been experiencing significant benefits from the tool, and have them share their experiences with everyone else.
Happy automating
Automating effectively isn’t easy, but with a clear strategy – and strong change management and technical design – fulfilling the promise of automation is within reach for any organization.