Automating Canvas Object Manipulation in KuJiaLe Tools with Puppeteer

Features of KuJiaLe Tools

KuJiaLe tool products are primarily HTML‑based web front‑end pages, and objects on the Canvas are not DOM elements, so they cannot be selected or manipulated via CSS or XPath selectors.

How the Automation Framework Supports Front‑End Canvas Operations

Since direct selection of non‑DOM objects is impossible and coordinate recording or image matching is unstable, the solution relies on the product itself. The KAF/APP Core framework provides APIs such as obtaining screen coordinates, spatial coordinates, size attributes, and camera positions.

We built an automation framework with Puppeteer as the core library to support these canvas operations.

Supported APIs

Press ‘~’ key to print Entity info in console – used for constructing test data (e.g., "scene-3d", "furniture", "738340").

getSelectedEntityComponent – retrieve properties of the currently selected Entity.

window.getPosition2DById / window.getPosition3DById – obtain the screen coordinates of an Entity (wrapped as kafCommon.getObjPosition()).

window.__dev_kaf.getEntityComponent – get Entity attributes (wrapped as kafCommon.getObjState()).

TODO: retrieve three‑axis or translation controls for the selected Entity.

window.__dev_kaf.getScene2dAttrs / window.__dev_kaf.getCameraAttrs – get camera position (wrapped as kafCommon.getCamera()).

window.__dev_kaf.setScene2dAttrsFromString / window.__dev_kaf.setCameraAttrsFromString – set camera position (wrapped as kafCommon.setCamera()).

window.getStateManagerChildrenAndState – get specific node attributes of StateManager (wrapped as kafCommon.getStateManager()).

window.kafProfiler.getFPS – obtain current canvas FPS (wrapped as kafCommon.getFPS()).

window.kafProfiler.dump – retrieve canvas tile data (wrapped as afCommon.getComponent()).

Demo: Moving the Bedroom Bed

The demo walks through the steps to move a bed in a design:

Open the design.

Switch camera view to focus on the target object.

Get the object's screen coordinates and select it.

Retrieve the object's initial XYZ spatial coordinates.

Move the object.

Retrieve the final XYZ coordinates.

Compare start and end coordinates to verify the move.

Images illustrate each step, the test data, and the execution results.