Implementing a Cross‑Platform Mobile UI Inspector with Flask and jQuery

Continuing from the previous post about the Nico automation tool, this article briefly describes how the author created an iOS and Android inspector that displays the UI hierarchy and a live device screenshot in a web page.

To keep the implementation simple, the whole UI is rendered on a web front‑end while Flask is used as a one‑file back‑end to serve images and the UI‑XML. Flask proves to be a handy choice for small web utilities.

The page layout consists of three sections: the element tree on the top‑left, element details on the bottom‑left, and the device screen on the right.

When the inspector starts, it first requests the device server for a screenshot and the UI tree. The Flask routes that handle these requests are:

// Python Flask routes
@app.route('/refresh_image')
def refresh_image():
    port = int(os.environ.get('RemoteServerPort'))
    platform = os.environ.get('nico_ui_platform')
    if platform == "android":
        new_data = send_tcp_request(port, "get_png_pic:100")
    else:
        new_data = send_tcp_request(port, "get_jpg_pic:1.0")
    base64_data = base64.b64encode(new_data)
    return base64_data

@app.route('/refresh_ui_xml')
def refresh_ui_xml():
    root = dump_ui_tree()
    html_list = xml_to_html_list(root)  # build HTML list
    return html_list

On the client side, jQuery periodically fetches the image and XML and updates the DOM:

// jQuery data refresh
function refreshData() {
    bounds_list = [];
    // refresh image
    $.get('/refresh_image', function(data) {
        var img = document.querySelector('img');
        img.src = 'data:image/png;base64,' + data;
        img.setAttribute("id", "image_");
    });
    // refresh UI XML
    $.get('/refresh_ui_xml', function(data) {
        var xmlContainer = document.querySelector('.content-inner');
        xmlContainer.innerHTML = data;
        initImageControl();
        addTextControlHoverListeners(); // add listeners
        addImageListeners(); // add listeners
    });
}

After obtaining the XML tree, the author recursively traverses all nodes and creates a div for each element. The conversion function is:

# Recursive conversion from XML to HTML list items
def xml_to_html_list(element, depth=0):
    random_number = random.randint(100000, 999999)
    html = f'
&
{content}
'
    if element.attrib:
        html += " (Attributes: "
        html += ", ".join([f'{k}="{v}"' for k, v in element.attrib.items()])
        html += ")"
    if element.text and element.text.strip():
        html += f" - Text: {element.text.strip()}"
    html += "
"
    children = list(element)
    if children:
        for child in children:
            html += xml_to_html_list(child, depth + 1)
    return html

Each generated div corresponds to a node in the UI hierarchy, and a transparent overlay div is added on the screenshot with the same random identifier. Hovering over a node highlights the matching area on the image and vice‑versa, providing a synchronized inspection experience.

The random identifier (e.g., nico_123456 ) is also used to give each text node a unique id , enabling the bidirectional highlighting between the tree and the image.

With these mechanisms, the inspector can display the UI tree, element details, and live device screen, and highlight corresponding elements on hover. The current version is read‑only, but the author plans to add click interactions in the future.

In the concluding remarks, the author reflects on the learning process, the value of open‑sourcing the tool, and encourages others to share technical knowledge despite the challenges of finding time for such contributions.