How AR Transforms Coffee Retail: Inside Alibaba’s AI‑Powered Cloud Recognition

AR Overview

AR (Augmented Reality) merges real and virtual images in real time. In the Starbucks Reserve Roastery, an AR assistant detects coffee pots, overlays interactive buttons, and tracks them throughout the brewing process.

Client‑Side Recognition

Client‑side recognition runs on the device using classic machine‑learning algorithms to identify large, texture‑rich objects such as copper vats and roasters. Pre‑processing strategies make the inference fast and stable without network latency.

The shop contains many varied objects, presenting three main challenges:

Metal reflection: appearance changes with viewing angle.

Transparency: containers may be empty or partially filled.

Dynamic environment: small items, diverse cameras, and changing lighting.

Cloud‑Based Deep‑Learning Recognition Service

To overcome client limitations, a cloud service based on deep learning was built. It combines two model types:

Object detection model – locates objects but has weak perception.

Image classification model – has high perception but cannot locate objects.

By fusing both, most objects are detected for position, while difficult‑to‑perceive items are classified and a guessed position is assigned.

Image Synthesis for Data Augmentation

Training deep models requires massive, diverse data, especially for reflective or transparent items like French presses. Image synthesis automates data generation by compositing target objects onto arbitrary backgrounds.

Capture many images of the target on a green screen.

Automatically remove the green background.

Blend the object with any background and adjust colors to match.

Image synthesis does not need physically plausible results; its goal is to teach the model foreground‑background separation.

Color and Imaging Simulation

AR perception also depends on camera imaging quality. Two automated algorithms simulate other cameras:

Pure color transfer – aligns the color distribution of one image to that of another.

Response‑curve simulation – replaces camera A’s response curve with camera B’s to generate a simulated image.

AR in New Retail

The AR system transforms the Starbucks Reserve Roastery into a new‑retail experience, allowing customers to explore the shop via their phones, discover hidden coffee‑making steps, and bridge online and offline interactions, demonstrating the limitless potential of AR‑driven retail.