During a product management meeting last week, I stated that “Performance is a feature, not a Bug.” Not every one got the meaning of that statement, so I will try to explain my thinking behind it.

Performance Breakdown:

Performance is the quintessential problem that costs more the later in the development cycle it is found to fix. Let’s look at some of the reasons:

Performance tuning is typically going to take some dedicated time and devotion to get as much performance out of the system as possible. Whom ever the engineer assigned to the task they will need to look at the system from both a component level as well as the system level to find all the performance issues. Along the way developing stress testing scenarios and code to put a load on the system to see where the bottlenecks exist.

In many large scale applications the performance issues may not be as transparent as; let’s optimize this SQL statement or change this function to faster. Large code base’s also have the problem of ramp up time. The resource assigned to the task may not know all the components of the system, for them trying to find the real issue will not be easy. Providing the engineer with tools will shorten the time considerably but will also come with a learning curve for the tool itself.

Many times once performance is found the solution isn’t very eloquent. Back in the day, I was developing a high performance real-time graphics engine for a piece of software that overlaid images on top of digital video. After tuning as much of c++ code as possible, I still needed way more performance out of the application. I had to turn to assembly and write directly into the Intel MMX registers allowing me to perform the four RGBA operations simultaneously. In the end, the system performed better than needed, which means it worked as expected. Starting out, I knew a small amount of assembly but needed to use the MMX extensions which were new. This incurred ramp up time. Also now within the software was a section of code that was outside the main development language requiring advanced knowledge and possible issues.

Even worse than a brute force solution, some performance fixes require a re-design or re-architecture. Even further back in the day, during the first product release I was ever involved in. SQA found during testing there was a major performance issue in a piece of Network management software I was working on. The issue only occurred in the testing lab which made it even worse to diagnose. Though the issue ended up being a configuration problem on the testing machine. The software still needed to handle the problem and work smoothly which took sometime to implement. In the ended, I had to re-engineer a low level component which introduced a stability risk that needed to be worked out through testing.

To sum up what it takes to get performance right:

• A dedicated resource and time allocation in the development cycle
• Understanding of the all the components with in the system as well as the sub-system that the application is being built on.
• Ability to get down and dirty and use of brute force techniques rather than an eloquent one.
• Tools to find the problem areas in the code base. Issues may not be where you think they are.
• Development of the stress testing scenarios and code to stress test at both the component and system level.
• Issues way be environmental.

Performance as a bug

So what happens if a person is not dedicated to finding and fixing performance as part of the release cycle? Typically everything has the potential to cause performance issues, so SQA or customers are bound to find issues with in the product. Given then the finding of an issue will either occur in the testing phase or after shipment, only a limited time will be allotted to find and fix the issue. Otherwise the release will need to be delayed or the customer will grow tired of the product and find something else.

To deliver on schedule, the team will typically use the application that produces the performance problem as the benchmark for its resolution. The resolution can occur one of two ways, the application is fixed by changing the application to work around the problem. This really doesn’t fix the issue just gets the release out, or the issue is fixed in the system. This is a better solution but only address the certain issue in the system and not the performance of the overall system as a whole. Typically this solution means the customer just finds the next performance issue.

Performance as a feature

So if performance is going to take time and devotion to get right, waiting to the end of a release to look into performance is probably not the best time to address it. You will need to procure and learn a new tool, get a deep understanding of the sub-system and possibly write a bunch of new code that changes a core piece of the product. None of the aforementioned items is a task that someone should perform at the end of a release.

Requirements drive the direction of products, each requirement typically gets a resource and time allocation to meet the requirement. The resource then can setup a project plan and take a holistic look at the components and sub-system of the product. By doing it this way, test applications can be developed to stress the complete system rather than a one-off that tests only a certain aspect of the product.

Sum it up

No matter which way it gets done, it will take a lot of work to get the performance issues of a system worked out. But waiting until the end only means increased pressure on the team and limited resolution of the performance of the complete system.