WhereAreYou: Mobile AR App for Vehicle Finding Using Core ML and Multi‑CNN Detection

The article presents a detailed case study of a 2017 Meituan‑Dianping Hackathon project called WhereAreYou , an AR‑enabled iOS application that helps users locate ride‑hailing vehicles in low‑light conditions.

Motivated by the observation that many cars flash hazard lights after work hours, the team explored whether Apple’s newly released Core ML framework could be combined with ARKit to create a “cool” product that visualizes the direction and distance of nearby cars.

The system architecture (Figure 1) consists of a simple Node.js backend that manages shared‑location groups. Users join a group via a group ID, and the server distributes each member’s GPS coordinates. An AR mode opens the camera; when another group member is nearby, a 3D label appears on the screen indicating the person’s name and distance.

Mapping GPS coordinates to screen positions required solving a geometric problem: given two points P0 and P1 (latitude as X, longitude as Y), compute the north‑bearing angle θ; obtain the device’s compass heading α from the gyroscope; then render the 3D model at the screen‑center offset γ = α − θ. The article provides the relevant formulas and a Swift code snippet that updates the ARSCNView node positions:

func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
    guard let renderLocations = self.netGroupInfo?.locations?.filter({ (userLocation) -> Bool in
        return userLocation.userId != GroupMenberManager.sharedInstance.getCurrentUserID()
    }) else { return }
    DispatchQueue.main.async {
        guard let camera = self.sceneView.pointOfView else { return }
        let currentLocation = UserLocation()
        currentLocation.latitude = GroupMenberManager.sharedInstance.userLatitude
        currentLocation.longitude = GroupMenberManager.sharedInstance.userLongitute
        for renderLocation in renderLocations {
            let distance = currentLocation.distanceFrom(renderLocation)
            let angle = currentLocation.angleFrom(renderLocation)
            var position = SCNVector3(x: 0, y: 0, z: -3).roateInHorizontalPlaneBy(angle: self.compassAngle - angle)
            position = camera.convertPosition(position, to: nil)
            position.y = 0
            self.virtualObjectManager.findVirtualObject(renderLocation.userId ?? "")?.scnNode.position = position
            self.virtualObjectManager.findVirtualObject(renderLocation.userId ?? "")?.changeNodeTextAnSize(text: renderLocation.userTitle, distance: distance)
        }
    }
}

During real‑device testing the team discovered that GPS accuracy degrades sharply within a 10 m radius, causing the AR markers to jitter. To overcome this, they switched to a vision‑based solution that uses AI algorithms to detect and track vehicles directly from the camera feed.

Using Core ML, the authors converted several open‑source CNN models (e.g., Caffe’s web_car model) into .mlmodel files with coremltools. Example conversion script:

import coremltools
coreml_model = coremltools.converters.caffe.convert(('web_car.caffemodel', 'deploy.prototxt'), image_input_names='data', class_labels='class_labels.txt')
coreml_model.save('CarRecognition.mlmodel')

The final pipeline combines three stages:

Vehicle detection using a YOLO‑style SSD model to locate cars in each camera frame.

Vehicle recognition by feeding the cropped car image to a CNN that classifies the make/model.

Vehicle tracking using Core ML’s built‑in tracking module to maintain a stable label across frames.

Figures 2–13 (included in the original article) illustrate the AR view, detection results, and tracking performance. A timing table (Figure 14) shows the per‑module execution time, confirming that the solution runs in real time on iOS devices.

In the conclusion, the authors acknowledge that the project relies on pre‑trained models without custom training, and suggest future work such as adding color or license‑plate detection, and improving robustness under adverse weather conditions.

References to related work (large‑scale car datasets, YOLO 9000, Apple’s machine‑learning documentation, coremltools, and the Caffe Model Zoo) are listed at the end of the article.