Since the dawn of computers, we’ve always had to test software. Over the course of several decades, the discipline of software testing has seen many best practices and patterns. Unfortunately, there are also several anti patterns that are present in many companies.

An anti pattern is a pattern of activities that tries to solve a certain problem but is actually counter-productive. It either doesn’t solve the problem, makes it worse, or creates new problems. In this article, I’ll sum up some common testing anti patterns.

Only Involving Testers Afterwards

Many companies only involve the testers when the developers decide a feature is done. The requirements go to the developers, who change the code to implement the requested feature. The updated application is then “thrown over the wall” to the testers. They will then use the requirements to construct test cases. After going through the test cases, the testers will often find all sorts of issues so that the developers need to revisit the new features. This has a detrimental effect on productivity and morale.

Such an approach to testing is used in many companies, even those that talk about modern practices like Agile and DevOps. However, “throwing things over the wall” without input from the next step goes against the spirit of Agile and DevOps. The idea is to have all disciplines work together towards a common goal.

Testing is about getting feedback, regardless of whether it is automated testing or not. So of course you have to test after the feature has been developed. But that doesn’t mean you can’t involve your QA team earlier in the process.

Having testers involved in defining requirements, identifying use cases, and writing tests is a way to catch edge cases early and leads to quality tests.

Not Automating When You Can

Tests that run by the click of a button are a huge time saver, and as such they also save money. Any sufficiently large application can have hundreds or even thousands of automated tests. You can’t achieve efficient software delivery if you’re testing all this manually. It would simply take too much time.

One alternative I’ve seen is to stop testing finished features. But due to the nature of software, existing features that used to work can easily break because of a change to another feature. That’s why it pays off to keep verifying that what used to work still works now.

The better alternative to manual testing is to automate as many tests as you can. There are many tools to help you automate your tests. From the low level of separate pieces of code (unit tests) over the integration of these pieces (integration tests) to full-blown end-to-end tests.

As a tester, you should encourage the whole team to be involved in manual testing. It will encourage them to write code that is fit for automated tests. Help developers write and maintain automated tests. Help them identify test cases.

Expecting to Automate Everything

As a counterargument to my previous point, be wary of trying to automate every aspect of testing. Manual testing can still have its place in a world where everything is increasingly automated.

Some things could be too hard or too much work to automate. Other scenarios may be so rare that it isn’t worth automating, especially if the consequences of an issue are acceptable.

Another thing you can’t expect to automate is exploratory testing. Exploratory testing is where testers use their experience and creativity to test the application. This allows the testers to learn about the application and generate new tests from this process. Indeed, in the words of software engineering professor Cem Kaner, the idea behind exploratory testing is that “test-related learning, test design, test execution, and test result interpretation [are] mutually supportive activities that run in parallel throughout the project.”

Lack of Test Environment Management

Test Environment Management spans a broad range of activities. The idea is to provide and maintain a stable environment that can be used for testing.

Typically, we call such an environment a testing or staging environment. It’s the environment where testers or product owners can test the application and any new features that the developers have delivered.

However, if such an environment isn’t managed well, it can lead to a very inefficient software delivery process. Examples are:

●  Confusion over which features have already been deployed to the test environment.

●  The test environment is missing certain critical pieces or external integrations so that not everything can be tested.

●  The hardware differs significantly from the production environment.

●  The test environment isn’t configured correctly.

●  Lack of quality data to test with.

Such factors can lead to a back and forth discussion between testers, management, and developers. Bugs may go unnoticed or reported bugs may not be bugs at all. Use cases may be hard to test and bugs reported in production hard to reproduce.

Without a good test environment management, you will be wasting time and losing money.

Unsecured Test Data

Most applications need a set of data to test certain scenarios. Not all data is created equal though. With modern privacy laws, you want to avoid using real user data. Both developers and testers often have to dig in the data of the test environment to see what is causing certain behavior. This means reading what could be personally identifying information (PII). If this is data from real users, you might be violating certain laws.

Moreover, if your software integrates with other systems, the data may flow away from your system to a point where it is out of your control. Maybe even to another company. This is not something you want to do with real people’s data. Security breaches can lead to severe public image and financial losses or fines.

So you want either made up data or obfuscated / secured data. But you also want to make sure that the data is still relevant and valid in the context of your application. One possible solution to this is to generate the data your tests need as part of your tests.

Not Teaching Developers

The whole team owns the quality of the software. Pair with developers and teach them the techniques so that they can test the features as they finish them.

This is especially important in teams that (aspire to) have a high level of agility. If you want to continuously deploy small features, the team will have to continuously test the application. This includes developers, instead of having them wait for the testers.

In such a case, the role of testers becomes more of a coaching role.

If testers and developers don’t work together closely, both will have negative feelings for each other. Developers will see the testers as a factor blocking them from moving fast. Testers will have little faith in the capacity of the developers to deliver quality software.

In fact, both are right. If the two groups don’t collaborate, precious time and effort will be lost in testing a feature, fixing bugs, and testing the feature again. If the developers know what will be tested, they can anticipate the different test cases and write the code accordingly. They might even automate the test cases, which is a win for testers and developers.

Streamline Your Testing!

The major theme in this article is one of collaboration. Testers and developers (and other disciplines) should work together so that the software can be tested with the least amount of effort. This leads to a more efficient testing process, fewer bugs, and a faster delivery cycle. Top that off with good test environment management (which is also a collaborative effort) and secure data, and you have a winning testing process.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.

Today’s IT organizations are busier than ever. They process more data, employ more people, and empower more businesses than at any other time in history. This growth in IT power and responsibility highlights the necessity that IT organizations build upon good processes. Many organizations turn to ITIL and IT service management to provide structure to their IT organization. While ITIL is a terrific framework for managing IT organizations, it’s not a silver bullet. Simply knowing about ITIL and using it to structure your IT organization isn’t enough to ensure success.

If you’re concerned that your IT service management processes might be failing your team or your business, read on. I’ve laid out five red flags that will help you detect if IT service management is failing.

#1: You’re Not Properly Scoping Changes

A common mistake among IT organizations is failing to set realistic targets for success of new processes. This can take several different forms. All of them are quite damaging to your business.

One form is scoping that may be insufficiently measurable. For instance, leadership doesn’t provide any specific targets but merely sets a goal that things will “get better.” A goal that relies on relative measures of success like “getting better” means measurement will be subjective. Subjective measurements involve the perception of stakeholders, which can be easily swayed by variations in day-to-day service. You don’t want the business to perceive your team as failing because the CTO’s laptop just happened to have a faulty hard drive the day before an organizational review of Service Management objectives.

Another form is scoping that’s too ambitious. An example might be an IT service manager setting a service level agreement that says you’ll resolve all incidents in one hour. That’s not a realistic timeline. Setting unrealistic timelines for employees degrades morale and makes those goals seem meaningless.

The opposite problem can also be trouble for an IT organization. It’s no good to set goals that won’t accomplish anything at all. Setting a goal that’s too loose means your organization won’t need to change to improve and will fail to provide value to the business.

#2: You’re Using the Wrong Tools

While ITIL is primarily focused around creating good processes for your IT organization, tooling is still very important. Regardless of your role in your business’s IT organization, you need the right information at the right time to do your job. High-quality IT service management software is regularly underrated as a part of a good IT service management implementation. It’s not just about getting the right information to the right people. It’s also about making sure that software is easy to use for business users. An effective IT service management implementation puts customers in a position to succeed, even when other parts of the IT organization are failing.

One way to identify failing tools is by looking for common pain points. Spend some time with key users of your IT service management software. Do they regularly have a hard time finding things? Is their time spent trying to make sure they don’t “mess up” the software? Does the software itself suffer regular outages?

If you suspect that your IT service management tools aren’t living up to their promises, you might want to check out a new platform like Enov8. You may find that you can easily cover gaps in your processes with software instead of painful changes on the process side.

#3: You’re Thinking About Incidents Wrong

One way IT service management systems regularly fail their users is by focusing too much on fixing problems. I know, that seems like an odd response. The truth is that sometimes IT organizations can focus too much on fixing their own problems over solving problems for the business.

IT organizations regularly think about incidents as engineering problems while users think about them as an inability to get work done. I really like the analogy of a broken light bulb. An IT organization sees the broken light bulb as the problem. The business doesn’t see it that way. Instead, they feel that the problem is that they’re trying to work in the dark. Engineers might spend days trying to get a new light bulb to users while a much simpler fix would simply be to open the window shades.

IT service management works best when it focuses on delivering the results the business needs. Often times, that requires quality engineering, but it should never be the primary concern. If your team looks at a new incident and immediately jumps to figuring out the technical cause, your IT service management implementation is probably failing. Focus first on fixing the problem for the business before trying to fix the root cause.

#4: Your Processes Are Too Complicated

IT service management is about putting processes into place in order to solve problems for the business. This is a worthy goal! Unfortunately, lots of times organizations lose that vision in the day-to-day running of the team. Something goes wrong as part of incident response, so they add a new step to a process. That new step for the process solves one problem but creates another problem that isn’t immediately apparent. When a problem crops up from that new change, the team adds another step.

You can see where this is going. In trying to fix lots of little problems encountered by your IT service management implementation, you’ve created one big one. Your processes have become much too complicated. The consequences of over-complicated processes are numerous. Employees don’t know what to do while dealing with problems. Management can’t easily understand the state of any given incident’s response. The business is stuck suffering from open issues. Resist the urge to add a new part of the process every time you encounter a problem. If you’re in the habit of doing this, look for what steps of the process you can remove to simplify it.

#5: You’re Not Focused on People

ITIL books and training focus a lot on processes and systems. That’s necessary because people writing books or designing training don’t know the people in your business. But the truth of the matter is that those people are the reason for IT service management. The goal is to make their lives easier. It’s not about implementing a specific process or creating the perfect architecture.

At the end of the day, the true measure of success is whether your IT organization makes working for your business better. Successful IT service management implementations spend a lot of time thinking about their users. They talk with them and listen to the problems those users are facing. Unsuccessful implementations get bogged down by worrying about metrics and tweaks to the process.

The Hardest Part is Recognizing the Problem

Most IT service management implementations don’t fail because of malice. They don’t fail because of incompetence on the part of the team. Those implementations fail because the team didn’t recognize the warning signs of failure before it became entrenched within their system. The IT organizations pursued their implementation with the best of intentions but didn’t know they were headed toward failure. If you recognize some of these issues within your organization, it’s not too late to start fixing them. It’ll require diligence and critical thinking, but you can absolutely be successful.

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.

Key TEM Vendors

• Apwide Golive
• Enov8
• Omnium
• Plutora
• ServiceNow
• Xebia (Digital.AI)

Key TEM Performance Areas

1. Modelling
2. Booking Management
3. Coordination
4. Ticketing
5. Health Monitoring
6. Automation & DevOps
7. Data Management
8. Reporting
9. Extensibility
10. Affordability

Area-1 Environment Modelling

The ability to know what your Environments and Systems look like.

Historically think Visio or your CMDB (if you have one).

Gold Medal Position:   

Enov8 & ServiceNow both offer powerful Visual CMDBs & Component / discovery mapping.

Silver:

Plutora & Xebia offer modelling capability.

Bronze:

Apwide & Omnium modelling is achieved via tabular forms.

Area-2 Booking & Contention Management

The ability to capture environment requirements & manage contention on Environments & Systems.

Historically think Email & an attached Word document.

Gold Medal Position:   

Enov8 & Plutora offer advanced booking & contention analysis methods.

Silver:

Apwide, ServiceNow offer booking requests (ref ticketing) capability.

Bronze:

Xebia has no obvious environment booking or contention mechanism.

Area-3 Environment Coordination

Tracking Events & Release activity across space (Environments) & time (Month, Year etc).

Historically think a MS Project Plans.

Gold Medal Position:   

Apwide, Enov8, Plutora, ServiceNow offer Environment & Release based calendaring.

Note: Enov8 & Plutora offer Runsheets /Implementation Plans (respectively).

Service Now offers checklists.

Silver:

Xebia – Calendaring is release centric (opposed to environment centric).

Bronze:

Omnium (limited capability identified).

Area-4 Ticketing

Ticketing / IT Service Management to capture Environment Change Requests Incidents etc.

Historically think Remedy.

Gold Medal Position:   

ServiceNow has advanced ITSM methods.

Silver:

Apwide (using Jira), Enov8, Plutora have solid Ticketing / Requests functionality.

Bronze:

Omnium & Xebia dependent on other tools.

Area-5 Health Monitoring

The ability check Systems or Components or Interfaces are up.

Historically think Test Automation scripting or your server monitoring solutions like Zabbix.

Gold Medal Position:   

Enov8 & ServiceNow offer integration methods & native agents to monitor health.

Silver

Apwide & Plutora have APIs that logically allow system health updates.

Bronze:

Omnium & Xebia don’t play in this space.

Area-6 Automation & DevOps

The ability to automate key Environment Operations using code.

Think Jenkins or Puppet Jobs.

Gold Medal Position:   

Xebia is a powerful release orchestrator (its primary purpose).

Silver:

ServiceNow Orchestration automates IT & Business Processes.

Enov8 offers “agnostic” Scripting Hub (Orchestration Manager), Pipelines, Playbook, Webhooks & URL Triggers.

Bronze:

Apwide integration is very simple but can be achieved with Get/Post methods.

Plutora needs other tools to automate/integrate properly (like Dell Boomi). The SaaS only option can also be limiting.

Omnium integrates with other tools to automate.

Area-7 Data Management

The ability to manage one’s data e.g. Extract Data, Masking data, Provisioning Data etc.

Think Compuware File-Aid.

Gold Medal Position:   

Enov8 seems to be the only solution for Test Data. Enov8 offers support for Data (PII/Risk) Profiling & Masking and Data Bookings through “Data Compliance Suite”. Enov8’s Visual Orchestrate can also be used to schedule other Data Tools.

Silver:

Xebia & ServiceNow capabilities are limited but they can leverage their orchestrators and call other tools.

Bronze:

Apwide, Omnium & Plutora don’t appear to play in this space.

Area-8 Reporting

The ability to get & share insights about your Environments.

Historically think drawing pretty pictures & graphs with PowerPoint.

Gold Medal Position:   

A lot of the tools have solid reporting; however, focus is Environments: No Gold Medal yet.

Silver:

Enov8 seem to have best out-of-box Environment dashboards. Needs simpler customization.

ServiceNow Env Dashboard are limited but ultimately extensible.

Xebia have some solid report, but more deployment focused.

Plutora is reliant on a new “Tableau” extension. Getting there but seems disjoint.

Bronze:

Apwide leverages Jira’s native capabilities.

Omnium approach is somewhat “download/export” focused.

Area-9 Extensibility

The ability to have the product do whatever you want.

Think of Salesforce or SAP.

Gold Medal Position:   

ServiceNow – An Extensible Engine. You can use it to build anything.

Enov8 – An “Object Oriented” Extensible Engine. You can use it to build anything.

Silver:

Plutora has broad customization features so you can “partially” alter its behaviour.

Bronze:

Xebia allows customization of your processes but not the platform itself.

With Apwide & Omnium you basically get what you get.

Area-10 Cost

The money ball question. And potentially the most important for some.

Gold Medal Position:   

Low Cost of Entry – Apwide, Enov8 (Free Team Edition) & Omnium

Silver:

Medium – Plutora & Xebia

Bronze:

Expensive – ServiceNow (Just add another “0” for licensing & tailoring services)

The “Test Environment Management Tool” Score Card