How I Automated a Daily Dashboard Report with Node.js, Scraping, and DingTalk

Background

Our team used a third‑party system that provided a custom data dashboard. Every day we had to copy dozens of metrics from the dashboard into Excel, convert the sheet to an image, and post it to a DingTalk group. The manual process took 7–10 minutes per day and was error‑prone.

Process & Task Decomposition

We mapped the manual workflow to four automated tasks:

Automate login and obtain a token (or cookie) via reverse‑engineered JavaScript encryption.

Scrape the required data from the dashboard’s API endpoints.

Render the data as a table on a canvas and export it as an image.

Upload the image to OSS and send a markdown message with the image link through DingTalk’s webhook.

1. Reverse‑Engineered Login

The system uses a combined RSA + AES encryption scheme. By setting breakpoints in the login flow we located the passport/login call, discovered the isEncrypted flag, and traced the encryption logic to a method that first encrypts a short random key with RSA, then encrypts the login payload with AES.

Key points of the encryption:

RSA protects the AES key (short‑key length avoids RSA size limits).

AES‑CBC with PKCS7 padding secures the actual credentials.

Additional headers ( aes-salt, aes-iv, aes-passphrase, X-Request‑Timestamp, X-Request‑Id, X-Request‑Sign) must be reproduced exactly.

var b = require("crypto-js");
var jsencrypt = require("nodejs-jsencrypt/bin/jsencrypt").default;
/**
 * Simulate the JS‑derived login encryption.
 * @param body Login parameters
 * @param public RSA public key
 * @returns {{headers: Object, body: string}}
 */
function encryptLogin(body, public) {
  const W = new jsencrypt();
  W.setPublicKey(public);
  const q = (Math.floor(Math.random() * 1e6) + Date.now()).toString();
  const re = W.encrypt(q);
  const ie = b.lib.WordArray.random(128/8);
  const fe = b.lib.WordArray.random(128/8);
  const ue = b.PBKDF2(q, ie, {keySize: 128/32, iterations: 100});
  const ye = b.AES.encrypt(JSON.stringify(body), ue, {iv: fe, mode: b.mode.CBC, padding: b.pad.Pkcs7});
  const j = "/api/v2/auth/login?is_global=true";
  const Ee = parseInt(Date.now()/1000).toString();
  const he = Ee;
  const Fe = ye.toString();
  const bt = `${Ee}_${he}_${j}_${Fe}_14skjh`;
  return {
    headers: {
      "aes-salt": ie.toString(),
      "aes-iv": fe.toString(),
      "aes-passphrase": re,
      "X-Request-Timestamp": Ee,
      "X-Request-Id": he,
      "X-Request-Sign": b.MD5(bt).toString()
    },
    body: ye.toString()
  };
}

After encrypting, we POST the payload, capture the set‑cookie header, and reuse the cookie for subsequent API calls.

2. Data Scraping

We first inspected the dashboard to determine whether the data were rendered server‑side or fetched via XHR. The metrics were loaded asynchronously, so we captured the API calls in the network panel and identified the list endpoint and the detail endpoint for each metric.

Sample code for fetching the list and then each item’s details:

/**
 * Get dashboard list and details.
 * @returns {Promise<*[]>}
 */
async function queryReportList(dashboard) {
  const {id: dashboard_id, common_event_filter} = dashboard;
  const data = await fetch(`https://xxx/api/v2/sa/dashboards/${dashboard_id}?is_visit_record=true`, {
    credentials: "include",
    headers: {"User-Agent": "Mozilla/5.0 ...", Cookie},
    referrer: `https://xxx/dashboard/?dash_type=lego&id=${dashboard_id}&project=1&product=sensors_analysis`,
    method: "GET",
    mode: "cors"
  }).then(res => res.json());
  const result = [];
  for (const item of data.items.slice(0,13)) {
    if (item.bookmark) {
      const params = JSON.parse(item.bookmark.data);
      const res = await queryReportByTool({
        bookmarkid: item.bookmark.id,
        measures: params.measures,
        dashboard_id,
        common_event_filter
      });
      result.push({...res, name: item.bookmark.name});
    }
  }
  return result;
}

Running the script prints objects such as:

{
  name: 'xxx生成失败率',
  base_number: 0.0103,
  day: -0.3602,
  week: -0.1626
}
...

3. Image Generation

Using node‑canvas we draw a table on a canvas, then export it to a JPEG buffer.

4. Upload to OSS & DingTalk Webhook

We upload the image buffer to Tencent Cloud COS:

const filePath = `/custom/999/${dashboard.worksheetName}-${dayjs().format('YYYYMMDD')}.jpeg`;
const uploadRes = await tencentCos.upload(imageBuffer, filePath, true);

Then we send a markdown message with the image URL via DingTalk’s robot API:

async function sendDingTalkMessage(text) {
  const token = '1a6e1111111'; // robot token
  const result = await fetch(`https://oapi.dingtalk.com/robot/send?access_token=${token}`, {
    method: 'post',
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify({
      msgtype: 'markdown',
      markdown: {title: '监控日报', text},
      at: {isAtAll: true}
    })
  }).then(res => res.json());
  if (result.errcode === 0) console.log('发送成功');
  return result.errcode === 0;
}

After all four steps are verified locally, we schedule the script on a server (e.g., via cron) to run daily.

Key Takeaways

Start with a working manual flow, then replace each step with code (“first pass, then fill”).

Reverse‑engineering encrypted login requires breakpoint debugging and keyword search in the obfuscated bundle.

RSA + AES is a common pattern: RSA encrypts a short symmetric key, AES encrypts the payload.

Accurate replication of request headers (salt, iv, timestamps, signatures) is essential for the server to accept the simulated login.

Persist the obtained cookie for subsequent API calls.

Deploy the final script with a scheduler to achieve full automation.