Is your automation strategy helping you test better?

Every year countless automation projects are scrapped because teams miss on getting ROI with their automation projects. The reasons for scrapping automation projects are multiple yet one of the prominent reason I have seen is poorly strategizing automation.
Is your automation strategy helping you test better?

Introduction - Everyone wants to automate their Testing

Every time someone from leadership position understands the testing for software that they are developing is being done by humans and their are lot of mechanical repetitive activities being done by humans, spending lot of bandwidth, the first push comes from their side is can we automate this. Particularly when it comes to testing there are lot of activities like test data creation, tests infrastructure provisioning etc can be automated. I have often seen push for automating things always comes from top to bottom. For Testers working on ground it just gets added as another goal in their job and this is when the race begins.Most of the times testers without understanding what needs to be automated, at what level we should automate, till what extent we should automate and what to automate when straightaway jump on implementation and first thing they start with is creating framework. Without having proper answers to many of such questions often addressed by automation strategy they start automation. While automation can serve tremendous value in testing process poorly strategize automation feels like a burden or missed on adding value to overall testing. So the crux is everyone wants to automate but really few know how to make your automation effective. With this blog we will try to understand reasons for same and I will try my best to throw some light on how you can make your automation really effective.

The current state of Automation Projects in Industry

As of 2025 I see their are few popular tools and libraries in the market, please excuse me if i miss on mentioning your favorite tool since what we are talking here in this whole blog is really strategy and hardly anything about tools.

  1. - Selenium
  2. - Playwright
  3. - Cypress
  4. - RestAssured
  5. - Cucumber
  6. - Robot framework
  7. - Appium

As of today their are following popular patterns often seen in automation projects

  1. - Page Object Pattern
  2. - Repository Pattern
  3. - Singleton, Builder and other popular patterns

Yet we see almost 80% of every automation project are getting scrapped because stakeholders don't see those adding any value and don't get optimum Return on Investment with those. This trend is really concerning. It simply means industry is wasting lot of efforts on test automation although intent to start any of such projects were always in good faith of project. We have good tools in market some open-source some paid. We have almost many things technically feasible. Yet why such trends. Lets see the reasons.

Reasons why most automation projects fails

Most of the times people don't find value with poorly implemented automation because of following reasons

At least one of my test from total tests always fail for invalid reasons than application bug

Many implement test automation at various levels. But the major trend I have seen is most of the automation is implemented and run on broader stack. When we add too many things as dependency for automation chances of going something wrong increases depending on how broad the stack is. There is one more reason why tests fail for invalid reason is simply because they are flaky. The tests are often racing against the application speed and application development lifecycle any issues with synchronizing at both this places can cause false failures. Tests which fail for invalid reason add unnecessary noise in overall feedback your automation is giving you about software. Often due to this noise automation failures are always overlooked and feedback is lost.

We want to automate everything

Not just hundreds but I have seen often thousands of tests are being automated by software testers and developers for software projects. Its most of the time push from leadership to automate everything. What are we chasing is code coverage and volume in such scenarios most of the time. What we really need to chess is Impact and Risk Coverage. The problem with automate everything is we are chasing quantity not quality for automation. The efforts required to automate are also huge and often in the pile of too many things important things gets missed.

Only testers as consumer and creators of automation

In many of the projects. Testers or QAs are only people taking care of automation. Developers and other roles hardly care about adding anything or looking at feedback provided by automation proactively. Most of the times issues reported by automation or issues reported in automation are first looked by Testers or QAs and then they are relayed to developers. Problem with this is your automation is not first class feedback mechanism for your products testing.

Automation growing as tail of Hanuman

In an effort to support continuous feature development and with the growth of feature sets in software. Your automation keeps on adding automated test cases. Often people miss on deleting things from automation as and when something is not adding any value to suite. We already talked about drawbacks of bulky automation. It becomes unmaintainable and unsupportable as time progress and eventually gets scrapped.

Solution filled with Spaghetti code

In an effort to add more shiny framework features automation solution often becomes Spaghetti of code. It becomes unmaintainable and hard to comprehend. People automatically pay less attention to adding value to code which they don't understand or is hard to comprehend.

Trying to automate everything at one specific layer

Many people understand automation as just UI automation. They don't strategies what to automate at which layer and when. We already talked about drawback of automating at broader stack. Problem with such approach is to run your automation it takes lot of time. I have seen automation Runs taking 3 hours 4 hours or at a times 12 hours as well. If your automation is taking more time to run you will run it less frequently too often missing early feedback in development cycle.

Reasons for poorly crafted automation strategies

Most of the time automation projects are started directly with end goal of automating testing. There is one fundamental aspect many miss is while a bunch of tests can be automated not all Testing can be automated. A well crafted automation strategy goes hand in hand with overall testing strategy. Reasons for poorly crafted automation can be seen as following

  1. - Missing to consider Test Data strategy
  2. - Missing to consider deployment cadence and different environments on which automation can run
  3. - Trying to automate everything at UI layer
  4. - Less or No collaboration with Dev and other team members to define what needs to be automated at where
  5. - Missing on making your software testable
  6. - Missing on considering context and progress plan as part of automation strategy
  7. - Missing to add continuation strategy
  8. - Getting sold to one or other automation approach and not changing it as per need of hour

Some obsessions in many automation approaches

The influencer material on test automation is floating widely on internet and most of the times we see following obsession as one size fit all fallacy.

Bulky Regressions without fitness routines

As we already talked people tend to automate not just hundreds but thousands of tests without making old tests obsolete. The obsession to automate as much as possible makes your automation bulky and needless to say takes more time to run.

Test Pyramids without Egyptians

I have worked with the teams in past who use test pyramid as guiding model for their automation. What I have seen is good teams rather than sticking to dogma pragmatically implement pyramid layers. There is high collaboration between QAs and Developers to discuss what can be covered at which layer. Team members conduct Test walkthrough after implementing test at any layer so that entire team is in sync. They are true Egyptians working on pyramid. Yet I see poorly implemented pyramid model just sticks to dogma. Devs and QAs work in silos. They stick to responsibilities as specific layers only. So in my opinion pyramid is good model but rip all benefits of it you need to work as Egyptians collaborating together.

Shiny new tools without raw material

The obsession with tools in automation industry is far concerning than any other thing in automation. Lets add this library. Let move our tests to XYZ tool. Lets generate reports using XYZ library and so many conversations happening without having raw material of testability as first thing to address in your software. Lets use this XYZ tool to figure out xpath for locators. Lets use auto-healing capabilities of XYZ tool to fix changing locators. In my opinion if you work on raw materials and base you hardly ever need to add any shiny new tool. A decent enough set of libraries and tool can still serve good automation purpose.

Upfront framework design

When automation projects are started people start with framework design as if it is first distinct phase in automation. Good design is design that is minimal and evolves with time. Yet I have seen people spending not just weeks but months to design initial framework. This upfront design often makes your framework rigid and you become prisoner of your framework. Rather than giving you flexibility your framework adds lot of constraints under which you need to work now.

Elements of good automation strategy

A good automation strategy is strategy that not only takes care about tools but end goals, context and moving time in consideration. Following things marks good automation strategy

  1. - What layers are we going to automate with which tools
  2. - When are we going to automate which layer
  3. - How are we going to collaborate on automation
  4. - With what frequency and triggers we are going to run tests on on which environments
  5. - How we define when are we going to mark which tests obsolete and remove
  6. - How are we going to address test data needs for automation
  7. - How are we going to take external dependency failures in account in scheme of overall failures

What suites when

Blindly following influencer advices about go with XYZ tool, go with Test Pyramid will not help you cater your automation strategy suitable for your context. Instead you should know what to use when

Lets talk about Test Pyramid

I like test pyramid. Its fantastic model when you are really caring about all layers. Its easier to implement this model on brand new projects where no software code is yet committed. But consider a software which is already built partially. There are no or very few tests added like unit ot integration layers. Will you still start with test pyramid model to implement automation? I won't. I will first try to build safety net around existing features that are running fine. Reason for this is retrofitting unit and integration tests to existing code is often complex job. You need to refactor lot and make your code testable first. What I would rather do in such situation is start with lowest possible layer may be API or UI where automation is practical. And come up with bunch of tests that cover risks associated with breaking existing functionality often called as characterization tests.

Why we forgot Swiss cheese?

In an effort to slim UI tests we forgot Swiss cheese model that was being discussed in past. The scenario which we discussed in previous point related to test pyramid can be often tackled with swiss cheese model. In Swiss cheese model you still have layers of tests but they act as filter rather than strict pyramid. Although it is complex to implement unit, component or integration tests in legacy software. We can still add some tests. Some parts of code might be still testable. We can consider these layers then as rather layers of swiss cheese than layers of pyramid. Whatever risk is covered with one layer can still serve purpose of making your tests fast and thin eventually. I would yet recommend swiss cheese as stop gap solution only and you should device a strategy on how to convert these cheese layers to pyramid layers eventually. We will talk about it.

And Ice cream cone also may sometimes serve purpose

A software where there are hardly any tests or no tests written at lower layer and lot of tests written at upper layer supported by lot of Human testing can still serve good purpose to start with. Its not bad situation to start with, but its bad to continue with timelessly. Such model is often called as Ice cream cone. You need continuation strategy to convert this ice cream cone to pyramid eventually.

It's more important how you keep continuing automation than how you start it.

As we discussed various about situations your software development might be happening, Its more important how you continue automation than how you start it.

Leverage what is at hand

Scrapping entire thing that you have in hand should be last resort that you should think. Your existing automation solution can be still leveraged by improving it with continuous efforts.

Shift down gradually

Suppose you have tests that are in Ice cream cone shape. What you can do is periodically conduct a review calls with developers and talk about which tests can be shifted down one layer at a time. An end to end test that just checks error message verbiage can be easily shifted to lower layer if not then api or integration layer at least. Periodically revisiting tests to identify and shifting them to lower layer can serve as good continuation strategy.

Revisit tests at fix cadence

Revisiting tests at fixed cadence becomes important in understand what impact and risks are being covered by tests at given point of time in software development lifecycle. As we already talked you can simply remove tests which are not required periodically. Either they are shifted to lower layers or those parts of functionalities are no more risk.

Conclusion

To conclude Automation which is blindly implemented without any strategy with poor strategy will not help you get maximum out of it. Considering Context, time, technically, tools and efforts while devising automation strategy can make your automation give good returns on investment. Implementing all automation at UI layer ultimately can be harmful yet starting with it can still serve good purpose. What you do initially matters less than how you continue automation efforts.