Cut & Paste Real Objects into Photoshop with AR in Under 10 Seconds

AR Cut & Paste Overview

Open‑source prototype that captures a real‑world object with a phone and inserts it as a new layer in Photoshop in under ten seconds.

System Architecture

Mobile Application – Source code in the /app folder of the GitHub repository; deployable to a smartphone.

Local Server – Bridges the mobile app and Photoshop, exposing a screen‑point interface and handling image data. Source code in the /server folder.

Object Detection / Background Removal Service – Currently an external HTTP service that runs the BASNet model for salient object detection.

Step 1: Configure Photoshop

Open Preferences → Plug‑ins , enable Remote Connection , and set a password. The password must match the value in server/src/ps.py; otherwise only an empty layer will be pasted. Use a non‑white background to improve SIFT matching.

Step 2: Deploy Saliency Service

Run the BASNet model as an HTTP service on a CUDA‑enabled GPU. If the service runs on the same machine as the local server, configure it to listen on a different port.

Step 3: Install and Run Local Server

virtualenv -p python3.7 venv
source venv/bin/activate
pip install -r requirements.txt
python src/main.py \
    --basnet_service_ip="http://X.X.X.X" \
    --basnet_service_host="basnet-http.default.example.com" \
    --photoshop_password 123456

The --basnet_service_host flag is optional and only needed when the service is behind an ingress such as Knative or Cloud Run.

Step 4: Install and Launch Mobile App

cd app
npm install
# Edit component/Server.tsx to set URL to the local‑server address, e.g. http://192.168.1.29:8080
npm start

Vision Models

BASNet (CVPR 2019) uses a two‑stage architecture: a dense‑supervision encoder‑decoder predicts a saliency map, followed by a Residual Refinement Module that learns the residual between prediction and ground truth.

For finer background removal the prototype can use U²‑Net (Pattern Recognition 2020), which adds multi‑scale residual refinement and outperforms many state‑of‑the‑art segmentation methods.

References

AR Cut & Paste repository (clipboard branch): https://github.com/cyrildiagne/ar-cutpaste/tree/clipboard U²‑Net implementation: https://github.com/NathanUA/U-2-Net BASNet paper:

http://openaccess.thecvf.com/content_CVPR_2019/papers/Qin_BASNet_Boundary-Aware_Salient_Object_Detection_CVPR_2019_paper.pdf