4 Ways to Run JavaScript from Python for Web Scraping

1. Introduction

During daily web crawling, parameters are often encrypted, so we need to analyze the page source. By debugging, we extract the critical JavaScript code and execute it with Python, obtaining the Python implementation before and after encryption.

This article discusses four ways to call JavaScript from Python.

2. Preparation

Write a simple JavaScript script to a file.

//norm.js
// calculate the sum of two numbers
function add(num1, num2) {
    return num1 + num2;
}

The file defines a function that adds two numbers.

3. Method One: PyExecJS

PyExecJS is the most commonly used method. It runs JavaScript code in a local JS environment (Node.js, PyV8, PhantomJS, Nashorn, etc.).

Install the PyExecJS package:

# py_exec_js_demo.py
# install dependency
pip3 install PyExecJS

Read the JS source from the file:

def js_from_file(file_name):
    """Read js file
    :return:
    """
    with open(file_name, 'r', encoding='UTF-8') as file:
        result = file.read()
    return result

Compile and load the JS string:

import execjs
from js_code import *
# compile JS string
context1 = execjs.compile(js_from_file('./norm.js'))

Call the JS function:

# call the add() method with arguments 2 and 3
result1 = context1.call("add", 2, 3)
print(result1)

Note: PyExecJS starts a local JS environment, which may be slower.

4. Method Two: js2py

js2py is a pure‑Python JavaScript interpreter that converts JS code to Python without a separate JS runtime.

Install the library:

pip3 install js2py

Create a context with EvalJs() and execute the script:

# create context
context = js2py.EvalJs()
# execute the whole JS code
context.execute(js_content)
# call the add function
result = context.add(1, 2)
print(result)

Note: Very long or obfuscated JS may cause conversion errors.

5. Method Three: Node.js

Use Python’s os.popen to run a Node command that executes the JS script.

First, ensure Node.js is installed and modify the JS script to export an init function:

// calculate the sum of two numbers
function add(num1, num2) {
    return num1 + num2;
}
// export function for Node
module.exports.init = function (arg1, arg2) {
    console.log(add(arg1, arg2));
};

Build the command string and execute it:

cmd = 'node -e "require(\"%s\").init(%s,%s)"' % ('./norm', 3, 5)
pipeline = os.popen(cmd)
result = pipeline.read()
print('Result is:', result)

6. Method Four: PyV8

PyV8 wraps Google’s V8 engine for Python. It does not depend on a local JS environment and runs quickly.

import PyV8
from js_code import js_from_file

with PyV8.JSContext() as ctx:
    ctx.eval(js_from_file('./norm.js'))
    # call the add function
    ctx.locals.add(1, 2)

Note: On macOS and Windows with Python 3, PyV8 often fails, so it is not recommended.

7. Conclusion

The article summarized four ways for Python to call JavaScript. In real crawling projects, it is common to first test the script with Node, then choose one of the first three methods for Python implementation.