Microsoft recently introduced its new Surface laptops integrated with Artificial Intelligence (AI), reigniting interest in AI PCs in the market. Besides Microsoft, other major PC manufacturers have also announced plans to release AI PCs integrated with Microsoft's new Copilot Plus. The AI PCs with system-on-a-chip (SoC) capabilities are designed to run generative AI tasks locally, also known as ‘on-device AI’. A forecast¹ shows that AI PCs with specific SoC capabilities designed to run generative AI tasks without accessing the cloud will grow from nearly 50 million units in 2024 (while the worldwide market is 250 million units) to more than 167 million in 2027.

What is on-device AI?

On-device AI refers to the implementation and execution of AI models directly on local devices such as smartphones, tablets, laptops, etc., without accessing remote servers. To deploy AI on these devices, the trained AI models are compressed to make them suitable for resource-constrained environments. The models are then typically converted to neural network graphs to ensure they are optimised for execution on specific hardware and AI frameworks.

Features of on-device AI

While cloud-based generative AI, represented by ChatGPT, has demonstrated impressive capabilities, on-device AI still holds unique potential due to its distinct features:

• Data privacy: With on-device AI, users can implement AI models without accessing remote servers and data is processed locally on the device. This eliminates the need to send sensitive information to the cloud, reducing the risk of data breaches and significantly enhancing user privacy and data security.
• Low latency: Since the processing occurs on the device, the time taken to transmit data between devices and remote servers is avoided. This feature provides faster response times, which is critical for real-time applications.
• Low operational cost: For cloud-based AI providers, increasing energy and bandwidth costs are vital factors for pricing. By eliminating data transport and transferring processing to local devices, AI service providers can offload part of the data centre’s energy consumption and reduce bandwidth costs.

Is on-device AI a game changer?

As on-device AI technologies develop, more digital devices will be integrated with real-time AI, allowing users to access AI solutions through smartphones, laptops, wearable devices, smart home devices, AR/VR headsets, etc. AI will be more accessible for individuals due to its advantages in privacy, latency, and offline functionality. This will greatly expand AI’s application scenarios from mainly business to daily life, significantly increasing the number of potential AI users and potentially changing the consumer electronics ecosystem.

Moreover, AI service providers will have more pricing flexibility due to the lower operational cost. By balancing AI services between on-device AI and cloud-based AI, providers can optimise their business models – using cloud-based AI for services requiring more scalability and complexity and on-device AI for services needing low latency and privacy. The current data centre model could be partially replaced by a more flexible business model, enabling smaller companies focused on specialised AI applications to gain more opportunities.

Companies benefitting from the on-device AI trend

• Leading chip manufacturers: On-device AI requires high-performance processors. Neural Processing Units (NPUs) are hardware accelerators designed to execute AI tasks efficiently. They play a crucial role in enhancing the performance and efficiency of on-device AI. The increasing demand for NPUs could benefit several chip manufacturers. Qualcomm is a leader in this area with its Snapdragon X Elite and X Plus processors, which are the only processors that meet Microsoft's performance requirements (>40 trillion operations per second) for deploying new Copilot Plus on laptops. Many laptop makers, including Lenovo, HP, and Dell, have chosen Qualcomm’s chips for their laptops that are integrated with new Copilot Plus. Although Intel and AMD are lagging behind Qualcomm in NPU performance, they are catching up. Intel plans to launch its next-generation processor, Lunar Lake, in Q3 2024, which is expected to compete with Qualcomm's current lead. AMD also plans to launch a processor with higher NPU performance later this year. The competition among these chip giants in the NPU space is worth monitoring.
• IP cores and EDA tool providers: Companies providing Intellectual Property (IP) cores and Electronic Design Automation (EDA) tools could also benefit from the on-device AI trend. EDA tools streamline the entire design process from initial concept to final implementation, ensuring that the NPU is efficient, reliable, and manufacturable. IP cores offer pre-designed, tested, and optimised components that accelerate development, enhance performance, and reduce risks. Synopsys and Cadence Design Systems are two leading companies in these areas and could be boosted by the competition in NPUs.
• Consumer electronics giants: Apple and Microsoft are pioneers in integrating on-device AI into their main products. Apple also designed its own NPU-accelerating processor, M4, to enhance product performance.

Conclusion

On-device AI is an emerging area poised to revolutionise AI applications. Its key features, including data privacy, low latency, and low operational cost, differentiate it from cloud-based AI, which holds the potential to transform the AI ecosystem. In the short term, this trend will benefit AI-related companies across semiconductors, AI electronics, and AI application providers. Looking further ahead, it will expand AI application scenarios in daily life, significantly lower the barrier for individual customers to access AI, and potentially make AI available to everyone.

¹ IDC: IDC Forecasts Artificial Intelligence PCs to Account for Nearly 60% of All PC Shipments by 2027.