This was a statement that I made as part of a question I posed at the ESL symposium. The gist of the question was that as software becomes a bigger part of the overall system it becomes impossible to fully validate the system without including the software. Hardware verification techniques are already beginning to become stressed and now we add in the software, that is not verified at all today and are we not heading for a train wreck? Needless to say the panelists took aim at my statement as being untrue.

Well, Wind River has just published the results from a survey they conducted that exonerates my position. Here are a few of their conclusions:

Survey respondents highlighted two realities that are clearly impacting their day-to-day operations. The first of these is the dramatic increase in the amount of software content (in terms of lines of code) that is now typically included in embedded devices. The software content is doubling about every two years. The sheer volume is making it increasingly difficult for QA and test teams to keep up with traditional tools and processes.

The second major issue is the rapid increase in architectural complexity. This includes moving from 16-bit to 32- and 64-bit architectures, greater utilization of multi-core technologies, and ongoing operating system changes. Other issues include the use of multiple operating systems in a single product, advanced user interfaces and networking, and virtualization technologies.

These newer systems and technologies pave the way for higher performance and greater capacity, but they significantly increase the overall complexity of embedded devices. With greater complexity comes the need for more and better testing.

So far so good. Nothing that was not already known right? So what are they doing about this added complexity?

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks increase the need for new approaches to test automation.

Or in other words most companies have no formalized way of doing verification, have no completion metrics and have little automation to help. So how bad are the metrics? Well take a gander at this:

When asked how they measure software quality today, survey respondents most often cited metrics that are reactive in nature such as tracking customer-reported failures and open defects rather than metrics that can help them prevent defects. These metrics, especially the customer-reported problems, count how many “horses have already left the barn.”

Here are a few more of their findings that I just can’t resist putting in:

Survey participants clearly indicated that they have a strong desire to be more proactive, but their responses also reveal a significant disparity between their intentions and their actual testing.

Respondents further reported that often the software quality information they do receive is later proved to be wrong, leading them to make product readiness decisions based on inaccurate information.

I rest my case. Software is not verified, it is only shown to work under a few typical conditions and the rest is left to luck. This has to change especially if it becomes a “System” responsibility.

-------------------------------------------
Brian Bailey – keeping you covered