Brought to you by Keysight Technologies
Todays’ newest products are increasingly complex, due to evolving standards, applications distributed across multiple cloud environments, and acceleration in an “all-in-one” ‘ expectation from new users. This increased complexity directly impacts development validation and test coverage for new products, increasing pressure on development teams.
With so many potential vectors to test, there is an expectation that automating design and test procedures would be mainstream.
However, according to a recent Forrester study commissioned by Keysight, today 89 per cent of companies still employ manual processes, with only 11 per cent of companies fully automating their test matrices today.
While full automation adoption is still low, companies do see value in automation, with 75 per cent reporting some automation and nearly half wanting to automate in the next three years.
With design and test automation poised to dramatically impact innovation cycles, we asked Jeff Harris, vice president for portfolio and global marketing at Keysight Technologies, to expand on the state of automation in the industry today and where automation will transform the development lifecycle.
Q: Why did Keysight commission a Forrester Test and Measurement Automation Thought Leadership Study?
A: In December 2021, we commissioned Forrester Consulting to evaluate the use of data integration, analytics, artificial intelligence (AI), and machine learning (ML) in a typical product development cycle. Forrester surveyed 406 development leaders and asked them a series of questions related to how much they currently employ AI and ML in their product development process.
The survey included developers across North America, EMEA, and APAC. We really wanted to understand the scalability of test innovation in a typical product development process.
Q: Are organizations satisfied with their current testing system?
A: On the surface, we were encouraged to hear that most organizations report they are satisfied with their current development approaches, with 86 per cent being moderately to very satisfied. When we looked underneath, though, we found that those same organisations report that 84 per cent of projects and designs are either complex, multilayer sub-systems or integrated systems, most of which are not being tested.
Despite what initially looks like satisfaction, we learned in the study that companies feel the pressure to do more, especially when asked about the future.
For example, in three years, nearly half (45 per cent) of companies would consider using a fully automated test approach, with 72 per cent expect to at least use augmented automation, where automation replaces some, but not all, of test processes, in their development process.
Currently, only one in 10 companies use full automation in their development process, but we expect the Covid pandemic to accelerate adoption of remote development, automated test sequencing. We also expect a much higher use of digital twins as development teams strive to continue working together but from different locations.
Q: Can you give an example of how companies can evolve to a more automated approach?
A: Here is a good example of where use of digital twin models would accelerate development processes and dramatically extend test coverage.
Any good software developer can emulate a sequence of test IP traffic. However, emulating a wide range of protocols across all the libraries of different user applications, and emulating traffic mixes of different users simultaneously exchanging different kinds of traffic at once, is much more complex.
At certain scale, it is easier for developers to use emulators, otherwise known as digital twins. Keysight offers more than 140 emulators creating digital twins for a wide range of channel and Layer 1 protocols, Layer 2-3 protocols, Layer 4-7 traffic, security, network application delivery, and more.
Even a traditional “C” programmer starting from a blank screen and adding low-level libraries proves problematic in a fast-changing development world. The complexity ramp-up is a new challenge requiring higher-level tools. Tools that take measurements and simulations into account as more systems and processes are being cloned into digital twins with deeper simulations and emulators.
Automation is rapidly becoming a must-have requirement. Currently, a fully manual test plan based on human data entry, some python or graphical programming, and Excel sheets can only cover a small portion of possible user stories. Each has to be updated manually for every software release.
New generations of software need to reflect application knowledge, advanced analytics algorithms, and measurement depth, wrapped in a framework of AI and ML to solve complex problems.
Q: How are the number of tests and test times increasing due to the complexity?
A: The introduction of multi-cloud environments, the still-evolving 5G standards, high-speed I/O protocols and intertwined application layering has increased the number of tests (77 per cent) and length of time to test (67 per cent) for companies.
For illustration purposes, let’s assume that a product used to be tested in 1 minute with 30 tests, and now, it takes 1 minute 45 seconds and over 50 tests.
While these numbers sound small, when both time and the number of tests increase at the same time, a developer’s validation matrix starts expanding exponentially rather than linearly.
The result would mean either increasing both validation and production tests, or cutting the reach of the test matrix. One choice dramatically increases unit costs, while the other choice dramatically increases operational risk. Neither is a good choice.
Q: How can companies react to nearly double the increase in test times and number of tests?
A: At Keysight, we see that while test automation software is part of the answer and definitely needed, it is not enough.
Automation is only as good as the analytics and insights they produce. In the Forrester survey, respondents disclosed that their test routines covered “more than needed” over half the time.
Automation can help reduce the time it takes to test, but doesn’t solve the test reach, quality, and coverage question. Automating intelligent, farther-reaching test sequences backed with analytics and insights would solve both test speed and test reach at the same time.
Q: What is Keysight doing to address these complex test issues?
A: Keysight created a model to describe how we combine deep measurement expertise with intelligent analytics and insights, and the knowledge of how to use those insights to automate workflows.
The end-to-end process is known as automating intelligence. These three core ingredients: measurement depth, analytics and insights, and workflow automation, are at the core of every piece of software Keysight delivers.
Automating Intelligence builds on the industry’s deepest measurement technologies and simulations to provide faster insights that developers can use to get to market with greater speed and lower risk. Whether measuring power and ground, waveform signal quality, high-speed data I/O’s, network integrity, or application delivery, we think about what it will take to help customers speed their development processes.
And as individual technologies become more interconnected and products have more required interfaces, Keysight will continue to help with:
• Automating intelligent insights across new workflows,
• Accelerating the engineering development approach, and
• Automating the necessary tests (leveraging advanced analytics, AI, and ML).
Q: How will automation help solve these challenges?
A: Keysight sees all this – the complexity, the reliability, the interactions, and the faster pace of new technology introduction –presenting a new development paradigm for companies wanting to push the boundaries of innovation: a need for farther-reaching automation and digital twin development environments to accelerate the speed of delivery and improve product quality.
Hardware developers have long relied on emulation environments as part of the layout before prototyping. Using emulators, or digital twins, reduces the number of design variables by allowing them to measure the impact of different operating environments, conditions, and protocol evolutions against known good references.
Similarly, software developers use scrum methods and test in emulation sandboxes to incrementally build and deploy new features in smaller groupings, while limiting the number of variables.
The rising complexity of product interactions – communication protocols that evolve, cloud platforms that evolve, continuous software and firmware updates – pose real challenges for developers as each represents a slew of new variables.
Using automation and continuously updated digital twins wherever possible enables development teams to reduce the variables related to their specific design. Reducing design variables using ‘known good’ digital twin references increases the likelihood that innovations that function in practice are the same as envisioned in the developer’s mind.
Q: How does intelligence factor in?
A: If you were building a new office building, you would want a top-ranked architect and contractor. You count on their intelligence in measuring every aspect and giving you thoughtful design choices.
Electronics are no different. The more precise the measurement system, the better the understanding of the team that builds your test system, the more you trust their results.
With the exponential increase in complexity and fragmentation, it’s necessary to move from simply ‘collecting data’ to making sure your measurement system has the intelligence you can rely on for better decision-making and insights.
Next-generation software supports this change by using advanced analytics based on an understanding of real applications and AI algorithms that continually process data, learn, and evolve to improve the decision quality. These actionable outputs and insights deliver better design choices for your development teams.
The advanced analytics that AI provides enable organisations to focus on what matters to users rather than trying to understand every aspect of the data.
Keysight’s intelligent automation spots any bugs or flaws that may exist. It proactively finds any weak points in a system and identifies what is critical to test from a user’s perspective. All of which provide a competitive advantage.
Q: Can an Automating Intelligence approach reduce time-to-market?
A: In the study we commissioned, Forrester identified that the three most significant contributors to time-to-market were shared data across teams, better analytics on current test and measurement data, and software tools across the product development lifecycle. Together, these factors can speed development and validation times, creating a faster time-to-market cycle. Organisations need to understand and address all three:
Data – Keysight believes that the next generation software will require improved data integration and sharing on all fronts:
• Physical – better measurement data and understanding of industry standards
• Digital – simulation and emulations, as well as combining physical and digital data from digital twins into a hybrid model with data fusion from several sources
Better analytics – Automation is a requirement for the future, but analytics have the potential to revolutionise test organisations with the advances of AI and ML paired with deep application knowledge to reveal true insights. “What to measure” is now a critical stumbling block for many as the permutations of complex parameters continue to grow at an unprecedented rate.
Software tools across the product development lifecycle – there’s a need for higher-level tools that can follow products, remove artificial barriers between design and test, and automate the workflow.
Q: What is the difference between Automating Intelligence and Artificial Intelligence?
A: Artificial Intelligence is only one building block needed to Automate Intelligence. How a measurement is taken, how it is analysed, and understanding how disjoint parts of a design connect are critical to intelligently automating any design, build, or operation.
While AI can successfully identify patterns, groups, relations, and outcomes, it is only as good as its designer in understanding the impact of changes in the industry standards, frequency bands, legislation, measurement limitations, customer use conditions, etc. This is why the more complete approach of automating intelligence is vital.
Q: What percent of companies are using AI today, and what will the rate of adoption be for AI?
A: We were pleased to see that the adoption rate of AI as a building block for Automating Intelligence is dramatically increasing. The Forrester study identified that 16 per cent of respondents currently use AI for integrating complex test suites. In the next three years, 52 per cent of companies surveyed (an increase of 325 per cent) reported considering using AI for integrating complex test suites.
Q: What is success? Are organisations ready?
A: The survey highlighted that for developers, the trifecta of automation and AI is comprised of increased productivity, the ability to simulate product function/performance, and bug fix automation/simulation.
In addition, there are wider business benefits stemming from adopting innovative test automation technologies that make up Keysight’s Automating Intelligence methodology. These include:
• A higher-quality product that increases customer satisfaction
• Ability to reduce product time-to-market
• More agile and efficient product development cycles
Development teams that have already adopted Keysight’s Automating Intelligence approach are reaping these benefits today. Some examples include:
• NASA uses Keysight’s intelligent test automation in the Orion space program to ensure that onboard software and equipment work as expected without faults. By deploying intelligent automation, NASA can accelerate the delivery and quality of its complex, mission-critical software system.
• Oxford University Hospitals NHS Foundation Trust is world-renowned for its excellence in healthcare, training, and research. It turned to Keysight to intelligently automate regression testing, populate training domains, automate appointment scheduling, and add patient observations. The reliability, predictability, and consistency of the automation have helped the hospital improve productivity and accelerate the pace of testing.
• Fujifilm Group turned to Keysight to automate testing of its software embedded in medical devices. The platform enabled it to achieve efficiencies and ensure a high quality and reliable product.
