Cliff Ortmeyer, Global Head of Technical Marketing at Farnell, considers the benefits of AI running at the edge of a network.

In a little over a decade since researchers uncovered novel techniques to improve its efficiency and effectiveness, deep learning has become a practical technology that now underpins a number of applications that need Artificial Intelligence (AI). Many of these applications are hosted in the cloud on powerful servers, as tasks sometimes involve the processing of data-rich sources such as images, videos and audio. Those servers often call on the additional performance of acceleration hardware, which range from graphics processing units to custom devices. This becomes particularly important for the numerically-intensive process, during which a neural network is trained on new data.

Typically, the inferencing process, which uses a trained network to assess new data, is far less compute-intensive than training. There are also workloads that involve less data-intensive sources, such as sensor readings from IoT devices, where both training and inferencing can be performed on lower-performance hardware. As a result, systems designers are finding that workloads do not necessarily need to be located in the cloud once the AI model has been trained, though many current services still do for business reasons. Instead, the trained model can be transferred to a local machine for processing closer to the source of the data.

The benefits of AI at the edge

There are a number of reasons for bringing AI models closer to the edge of the network. A major driver is privacy and user acceptance. For example, consumers using devices such as smart speakers are becoming increasingly concerned about having their private conversations routinely recorded and uploaded to cloud services for services that could be supported locally.

In industrial control applications, which are now beginning to use AI for machine-condition monitoring or process optimisation, concerns over the confidentiality of production data will similarly demand that as little as possible is transferred to the cloud. For many industrial applications, there are issues of the reliability and speed of the connection to the cloud. Many systems, whether on a shopfloor or situated in a remote location, do not have the high-bandwidth connections that are needed to support cloud-based inferencing.

Control systems are also adversely affected by high communications latency. If AI models are used within closed-loop control systems, any delay in fetching an update from the cloud will lead to inaccuracies and the generation of instability. Some systems may use a mixture of cloud and local processing. Surveillance cameras, for example, will preserve network bandwidth by identifying immediate threats locally but then call on the cloud to perform additional processing for situations with which the local models cannot cope.

Battery life is also significantly increased when AI sits at the edge, as far less data is sent over the network, reducing both network and cloud costs.

AI’s application in predictive maintenance

Predictive maintenance is currently the most popular use-case for connected-industry applications. It provides a high return on investment due to its ability to reduce the frequency of on-site inspections. Machine down-time can also be reduced by accurately identifying a component’s remaining useful time in service, making it possible to exploit its operational life without risking a failure during operation. Prediction of lifetime based on operational data also helps optimise maintenance schedules and accurately identify correct spares requirements. Customers using predictive maintenance are already seeing efficiency gains of 20 to 25 per cent.

With predictive maintenance, sensors in a machine tool may detect changes in temperature coupled with an increase in noise or vibration, picked up through a combination of microphones and accelerometers, as indicative of a potential problem. With traditional algorithmic techniques, it can be extremely difficult to relate combinations of real-time sensor readings to problems. AI-based models can interpret time-series data together with real-time inputs to determine the health of the system accurately. Such models can take advantage of techniques such as sensor fusion to determine whether the situation demands maintenance or not. The complexity and latency requirements of the model will determine the feasibility of running the model locally.

From Cloud to Gateway or Edge: approaches to local processing

There are two approaches to local processing. The first is to use the processing power available on the device itself. Whether this is feasible depends on the complexity of the AI model and the available processing power. For low-power sensor nodes, it will not be possible to run a large deep-learning model even for just inferencing, however, as machine-learning algorithms can take many forms, it may not be necessary to run such a heavyweight task completely locally.

The second option is to offload processing to another device either completely or partially. For example, the device may run a simplified AI model that performs an initial analysis of the data. In the case of a surveillance system that listens for activity, this model may simply perform the job of determining whether the signal is background noise, such as wind, or due to an intruder passing nearby or the sudden sound of a breaking window. More complex functions can be transferred to a gateway that concentrates data from multiple cameras and other security devices. The devices may provide some preprocessing to help streamline the job of the gateway model.

Powerful hardware such as the Xilinx programmable-logic devices provided by the Ultra96 platform from Avnet can perform inferencing based on sophisticated deep-learning models and other complex machine-learning algorithms. The local gateway may even update the model by collecting data and performing training in batches when the inferencing workload is light. Alternatively, the gateway can collect data that is important to retraining and pass that to a cloud server on a daily, weekly or monthly basis.

AI at the device provides the lowest communication latency. However, the faster processing speed of a local gateway may outpace a lower-performance device processor and offer the best latency and throughput parameters. Handoff to the cloud will rely on factors such as

available bandwidth on the internet connection and distance to the servers themselves: the finite speed of light imposes a limit on how low latency can be if the cloud is used for inferencing workloads.

Learning technology options

The developer has a choice of which types of model to employ for a specific application. Early AI models made use of structures such as decision trees or expert systems that demanded a large amount of effort from domain specialists to relate combinations of inputs to probable causes and outcomes.

Machine learning enhanced decision trees by making techniques such as random forests possible. Such an approach builds multiple decision trees from training data and applies them in parallel to compute an average that represents the most likely outcome based on the training data. A random forest is an example of a supervised learning system; it relates data provided by the system’s developers to the inputs from which the machine-learning model learns relationships.

Deep learning is another example of a supervised learning technology, as it relies on the training data being labelled beforehand. In an image-classification system, for example, the model takes in data from the image and applies it to a layer of simulated neurons. The outputs from the first layer of neurons are passed successively through many more layers. Some of them combine the outputs from multiple neurons in the preceding layer to produce a single value that is passed onto the next. In this way, deep neural networks perform dimension reduction: a vital step when converting complex multidimensional data such as imagery or audio into a form that can be used to provide a final classification. The neural network is able to perform classification because it can learn how different arrangements of coefficients in the neural layers respond to various images to provide outputs that match the training labels.

Training of a machine-learning model does not need to rely solely on labelled data. Unsupervised machine-learning algorithms such as clustering can find patterns in data without additional help. Such a process can be very helpful in industrial control systems where multiple sensors are being used or the time-series behaviour of the inputs is important. In the case of condition monitoring for a machine tool, the magnitude of vibration may not indicate a problem, it may simply be a consequence of the process. However, a pattern of movement in the time-series data coupled with rapid changes in temperature may be indicative of an issue that requires maintenance. Such differences can be revealed by the data separating into clusters that are easy to differentiate when the source data, if used directly, does not show a clear pattern.

An unsupervised machine-learning system can find patterns in the combinations of sensor readings that can be used to reduce the amount of data that is then fed to a supervised algorithm that is trained on machines operating under different types of stress and varying states of repair. As the clustering helps reduce the amount of data that needs to be transferred, one useful design pattern for implementation in IoT systems is to perform this activity on the device node. The model derived from supervised learning can run on the gateway or even in the cloud depending on the tradeoffs between latency, performance and cost.

Bringing it all together: AI to the edge

An important factor to consider with machine learning as a technology, is that its implementation often requires in-depth knowledge and experience in order to make the most of the different forms of AI that are available. Octonion has developed a software solution to this problem which is built into Avnet’s SmartEdge Agile device: allowing engineers to develop AI solutions for IoT systems that involve significant edge processing without hiring in expensive help or acquiring in-depth expertise themselves.

The solution runs at three levels: device; gateway; and cloud. The device hardware, the Avnet SmartEdge Agile, is an autonomous, low-power sensor node. This device provides access to a variety of sensor types that include accelerometer, gyroscope, magnetometer in addition to sensors to measure pressure, temperature, humidity and proximity. A microphone is available to record audio inputs.

The smart-sensor hardware communicates with a local gateway using BLE. Versions that support other IoT-oriented network technologies such as LoRaWAN, SigFox, cellular and WiFi are under development. The gateway module can take the form of an Android or iOS smart device; or alternatively, the gateway software can run on a Raspberry Pi or similar Linux platform device that can provide access to the cloud. The cloud layer can be deployed on AWS, Microsoft Azure or custom server solutions.

Coordination of the various layers is provided through Octonion’s Brainium software environment, which provides edge-to-cloud support in a zero-code environment. Security is key to all IoT deployment and an important element of the system in order to protect the privacy of the data being used to train models as well as during inferencing. The system developed for Brainium uses hardware AES encryption together with hardware-based tamper-resistant cryptography key storage to protect data as soon as it has been read by the MCU on the Avnet SmartEdge Agile device. The firmware images used by the devices to perform the AI functions are also encrypted, with digital signature validation. When data is sent, the communications channels employ TLS encryption to protect the messages from being snooped by adversaries from the edge into the cloud.

Octonion’s Brainium software environment provides a combination of unsupervised and supervised machine learning. For example, it can be trained on anomalies in the time series representation of raw sensor data. Additionally, it will cluster data from different scenarios to identify common data patterns. Those data patterns can be pulled into a flexible model builder within the AI Studio software so that developers can tune the model for the target use-case.

During the learning process, the device collects data samples, secures and encrypts them before relaying them to the AI Studio software running in the cloud. This software learns from the samples to generate AI models that it transmits back to the edge device for inferencing work. Any device within the customer’s deployment hardware can receive this AI model and operate in a standalone mode to perform monitoring and analysis. The environment that remains is consistent from prototyping through to full-scale production. For deployment, customers can use Avnet’s SmartEdge Agile hardware off the shelf or tune the design for use in their own implementations.

By harnessing smart data computation at the edge, the design of the Avnet SmartEdge Agile, powered by the Brainium software, minimises the volume of data that needs to be transmitted from each edge device. Where it is passed to the cloud, high security is built into the system. The result is a system that provides all the tools required for rapid development and deployment of AI-enabled systems.