The goal of every company is to satisfy its users. This certainly applies in the software industry. However, as the number of users increases, they tend to make more demands. Increased demands will increase how complex software is, as these demands may require adding new features. And of course, software firms try hard to control defects in their products whenever they add a new feature.

Nevertheless, the industry is still far from zero defects. To avoid defects in products shipped to users, firms in the software industry must pinpoint defects in their test environment before shipping products to users.

What’s a test environment, and how are developers making sure that they can find and cure defects in that environment? We’ll discuss both topics in this article.

When you’re running an IT organization, it’s not just the business that you have to take care of. One part of running a business is building, creating, and providing what your customers need. The other part is management. Out of all the things you have to manage, configurations and assets are two of the most important.

Although people often think of configuration management and asset management as the same thing, but they are different. People also sometimes confuse these terms with each other. So, in this post, I’ll explain what configuration management and asset management are and how they’re different. Let’s start by understanding each of these terms.

What Is Configuration Management?

Configuration management is the management of configuration items. So, what are configuration items?

Configuration Items

Any organization provides certain services. These services might be the ones being provided to customers or to internal users. Either way, creating and providing these services requires some components. So, any component that needs to be managed to deliver services is called a “configuration item.”

Too confusing? No worries—I’ll explain with an example. Consider that you’re providing a service that tracks an organization’s user data. In this case, you can consider the software to be the component that needs to be managed. It’s important that you manage this software to make sure your service works fine. This means that your software is a configuration item. Another way of defining a configuration item is that it’s a component that’s subject to change to make the service delivery better.

What Information Is to Be Managed?

When you manage the attributes of such configuration items, that’s configuration management. So, what kind of information do you have to manage? You have to manage attributes such as ownership, versioning, licensing, and types. Let’s consider an example in which you’re using software for internal tasks.

Now you’ve identified that the software that provides service is your configuration item. The next step is to manage information related to that software. The software developer will have released different versions of the software with updates and new features. You obviously look out for better versions of the software or the version that best suits your requirements. One piece of information that you have to manage is the details of the software versions.

Another example is when you’re using licensed software. The software will be licensed to a particular person or company, and the license will be valid for a certain period of time. Such information becomes the attribute you have to manage. Now that you know what configuration management is, let me tell you a little about how it’s done.

Configuration Management Database

An easy way to manage information on configuration items is by using a configuration management database (CMDB). A configuration management database is just like any other database that stores data, but it specifically stores information related to configuration items.

Configuration Management System

Configuration management isn’t easy. You have to take care of lots of tasks, such as tracking the data and adding and modifying configuration items. To make configuration management easy, you can use a configuration management system (CMS), which is software that helps you manage your configuration items. A typical CMS provides functions for storing and managing CI data, auditing configuration, making changes to the configurations, and so on.

Now that you know what configuration management is, let’s talk about asset management.

Asset Management

In generic terms, anything that’s useful is an asset. If you own a house or a property, that’s an asset for you. So is your car or your phone. When it comes to an organization, anything that’s useful to the organization is an asset. Assets can be capital, office property, the servers locked in your highly secured server room, and so on. But IT assets aren’t limited to physical or material things. The knowledge stored in your employees’ brains is also a valuable asset to your organization.

So, basically, tracking and managing the assets of your organization throughout its life cycle is asset management. The main aim of asset management is to create processes and strategies that help in managing assets properly. The asset management process starts right from the moment of acquiring the asset until disposing of the asset.

For example, let’s say you have an organization that builds and manages web applications. As part of this, you own some servers that you host the web applications on. You also have some databases where you store data for your clients. In this case, your asset management process starts from the time you bought the servers and the databases. You have to manage the buying, maintenance, and inventory costs. Along with that, you also have to take care of regular updates, audits, security implementations, and any changes that you make. This asset management goes on either until the assets are damaged or until they stop being useful to your organization and are disposed.

Asset management directly involves finance. You have to consider the inventory, governance, and regulatory compliance along with the financial aspects in asset management.

Why Do You Need Asset Management?

Asset management helps you understand your financial flow and how to efficiently plan your finances. You can easily track your asset throughout its life cycle. This helps you analyze incidents if something went wrong. Management of assets improves your assets’ quality and performance, which helps your business.

The asset management process helps you stay compliant with various rules and regulations. This improves the quality of your business and also saves you money on audits and fines. Because asset management lets you track your assets, you can plan more efficient strategies for operations.

Configuration Management vs. Asset Management

Now that I’ve explained each of these terms, I hope you understand what they mean. At some point, you might have felt that they were the same. To eliminate any lingering confusion, let me highlight the differences between them.

Asset management is managing anything valuable to your organization. You can consider configuration management to be part of asset management. Configuration management mainly focuses on managing configuration items and their attributes. These attributes mainly affect the delivery of the service.

In the case of asset management, it’s more of a financial perspective. You track the asset to understand the financial flow and need for that asset throughout its life cycle.

To understand the difference, let’s take an example of a hardware component that you’re using—let’s say, a database. When you’re using a database, the database itself becomes an asset. You have to manage the maintenance, track the asset, conduct audits, and so on. This is asset management. The same database will have software versions. Keeping track of the software version, updating it, and tracking which other components it works with becomes part of configuration management.

Configuration management and asset management might sound the same at a high level, but they have different purposes and are implemented differently. Understanding such terms with the help of an example really makes it easy to understand the differences, hopefully, the explanations and examples here have helped you.

This post was written by Omkar Hiremath. Omkar uses his BE in computer science to share theoretical and demo-based learning on various areas of technology, like ethical hacking, Python, blockchain, and Hadoop.

Having a set of test environments properly configured and managed is essential for modern software organizations. Creating and configuring such environments is part of a solid test environment management strategy. Unfortunately, as with many things in software development, this is easier said than done. There are many questions that need answering. For instance: how many test environments do I need?

 

The short, correct, but also totally frustrating answer is—you’ve guessed it—it depends. Like most things in our industry, there isn’t a one-size-fits-all solution.

 

This post features a longer, (hopefully) not frustrating version of the answer above. Answering “it depends” without explaining which things it depends on makes for a useless answer, so we won’t do that. Instead, we’ll cover the factors you have to take into account when making the decision on how many environments your organization needs. The most obvious one is probably organization size, but, as you’ll see, it’s not the only one.

Let’s begin.

What Are Test Environments?

Before we get into the factors we’ve mentioned, we have some explaining to do. Or, rather, some defining. In this section, we’ll define test environments. You’ll learn what they are and why do you need them.

Of course, if you’re already experienced in managing test environments—or have enough familiarity with the term—feel free to skip to the next section with a clear conscience.

A testing environment is a setup of software, hardware, and data that allows your testing professionals to execute test cases. For the test environment to be effective, you have to configure it, so it closely resembles the production environment.

As we’ve already covered, there are many types of test environments. Which ones your organization will need depends on several factors, such as the test cases itself, the type of the software under test, and many more. Since that’s the main topic of this post, we’ll get there in a minute.

But first, let’s quickly cover some of the main types of test environments available.

How Many Test Environments Do I Need? The Bare Minimum

We’re about to cover the main factors for deciding which and how many environments your organization should adopt. Before we get there, though, let’s talk about the bare minimum number of environments you need.

Development

The first obvious and indispensable one is the development environment. For some of you, it might sound weird to think of the dev environment as a testing environment, but it is. Developers should constantly test the code they write, not only manually (via building the application and performing quick manual tests) but also automatically, through unit tests.

You might consider the development environment an exception in the sense that, unlike most other environments, it doesn’t need to mimic production too closely. For instance, I have seen people argue that developers that create desktop apps shouldn’t use the best machines available. Instead, they should adopt computers that are close in configuration to those their clients use, so they can feel how the software is going to run. That’s nonsense. Developers should use the better and fastest machines their companies can afford, so their work is done most effectively. If performance is an issue, there should be a performance testing phase (and environment) to handle that.  The same goes for other characteristics of the production environment that don’t make sense for developers.

CI (Integration)

What I’m calling here the “CI environment” could also be simply called the test environment, or even integration test environment.

Here is the first step in the CI pipeline after the developer commits their code and push it to the server. The CI server builds the application, running whatever additional steps are adequate, such as documentation generation, version number bumping, and so on. Just building the code is already a type of test. It might help detect dependency issues, eliminating the “but it works on my machine!” problem.

If the application is successfully built, unit/integration tests are executed. This step is vital since it might be slow for developers to run all of the existing tests often in their environments. Instead, they might run only a subset of tests on their environments, and the CI server will take care of running the whole suite after each check-in/push.

QA

Then we have what we’ll call the QA environment. Here is where end-to-end tests are run, manually, automatically, or both. End-to-end testing, also called functional tests, are the types of tests that exercise the whole application, from the UI to the database and back again. This type of testing checks whether the integration between different modules of the software work, as well as the integrations between the software and external concerns, such as the database, network, and the filesystem. As such, it’s a really essential type of testing for most types of software.

Production

Finally, we have the production environment. For many years “testing in production” was seen as the worst sin of testing. Not anymore. Testing is production is not only forgivable but desirable. Practices like canary releases are vital for companies that deploy several times a day since it allows them to achieve shorter release cycles while keeping the high quality of the application. A/B testing can also be seen as a form of testing in production, and it’s essential for organizations that need to learn about their users’ experience when using their software. Finally, some forms of testing, like load testing, would be useless if performed on any environment other than production.

Which and How Many Environments Do You Need? Here Are the Criteria You Should Use to Decide

Having covered the bare minimum environments most organizations need, it’s time to move on. Now we’ll cover the main factors you need to weigh when deciding your testing approach. Let’s go.

Organization Size

The size of the organization matters when deciding which environments it needs. One of the ways this matter is in regards to personnel. Since larger companies have more people, they can afford to have entire teams or even departments dedicated to designing, performing, and maintaining certain types of testing, which includes taking care of the required environment.

Companies of different sizes also have different testing needs due to the software they create. It’s likely that larger companies produce more complex software, which would demand a larger pipeline. The inverse is also likely true for smaller companies.

Finally, organization size often correlates with the stage in which the company finds itself. That’s what we’re covering next.

Organization’s Life Phase

Do you remember when Facebook’s motto was, “Move fast and break things?”  It’s been a few years since they changed it to “Move fast, with stable infra.” While the new motto is definitely not as catchy as the previous one—some might say it’s even boring—it makes sense, given where the company stands now.

Startups have different testing needs than most established companies. Their priorities aren’t the same since they’re at very different points in their lifecycles.

For startups, beating their competitors to market might be more valuable than releasing flawless products. Established companies, on the other hand, will probably place “stability in the long term” way higher in the scale. They have their reputation at the stakes. If they’re public, they have to generate results for shareholders.

Therefore, more established companies will usually employ a testing strategy that adopts more environment, and it’s probably more expensive, and definitely slower. But such a strategy might give them the reassurance they need. On the other hand, startups that value time to market might choose a more streamlined pipeline, with fewer environments. Such an approach might be cheaper, easier to build and manage, but will give fewer guarantees than the more heavy-weight approach of the enterprise.

Software Type

The type of software developed is a huge factor when it comes to testing. A database-based web application with a rich user interface will require UI and end-to-end testing, for instance, while a library will not.

Similarly, user-acceptance testing makes sense for applications targeted at final users. For libraries and frameworks, unit and integration tests might suffice. You might have even more specific needs, such as integration with custom hardware, which can require more environments.

The type of software will dictate the required types of testing, which, in turn, will help you decide on the environments.

Domain or Industry

Some industries are highly regulated, while others are less regulated or non-regulated at all. That also has a huge impact on an organization’s testing approach. Domains like financial services and healthcare come to mind.

Your company might need to adhere to rules, regulations, or norms that govern whatever industry it operates in. That might require you have an additional environment in order to test that the product is compliant with these rules.

Time for the Verdict

So, based on all that we’ve just seen. How does one choose which test environments their organization needs? We’ll now, as promised in the title, offer you a quick recipe, or a step-by-step guide.

1. Start with the basics. Meaning, start with the bare minimum environments we’ve mentioned and then build upon it as your requirements change.
2. Consider the organization’s size and stage in life. Take into account the values and priorities of the organization (time to market vs. stability, disruption vs. market share, etc.), available personnel, and budget.
3. Take into account the type of software you make and the industry you belong to.

With that in mind, make your decision. If your organization makes a picture editing app for Android and iOs, you might want to have (besides the obvious dev and prod):

• The CI environment to perform unit and integration tests.
• A QA environment to help you with end-to-end/integration tests, using both emulation and real devices.
• An acceptance testing environment, where stakeholders give the final sign-off for the app’s release.

But if you’re creating a banking application, you could add an additional security and compliance environment. (Keep in mind that this is just an example. I’m not well-acquainted with the financial domain.)

Final Considerations

Test environment management is vital for the modern software delivery process. One of the decisions a test environment manager needs to make is how many environments to use. As you’ve seen, there is no one-size-fits-all answer, but that’s no reason to despair. There are objective criteria you should use to help you with your decision.

The journey isn’t easy, but this blog has many articles that can help you master test environment management and take your organization’s testing approach to new levels.

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.