How to Parse MDX Dictionary Files with Python and Export to CSV

Original data and processing results:

https://gitcode.net/as604049322/blog_data/-/tree/master/mdx

After downloading the help.mdx dictionary, we can read it with the readmdict library.

Install the library: pip install readmdict On Windows also install python-lzo: pip install python-lzo Example of reading the MDX file:

from readmdict import MDX

mdx_file = "help.mdx"
mdx = MDX(mdx_file, encoding='utf-8')
items = mdx.items()
for key, value in items:
    word = key.decode().strip()
    print(word, value.decode())
    break

The dictionary data appears as a JavaScript script; we can extract the JSON part with a regular expression:

import re, json

topic = json.loads(re.findall('"data":(\[.+\])}\);', value.decode())[0]

Example output for a simple entry:

[{'id': 'a', 'isroot': True, 'topic': 'a', 'describe': '英[ə; eɪ]美[ə; e]art. 一'}]

For words with hierarchical relationships, such as abalienate, the extracted structure looks like:

Our goal is to transform each word into a CSV row with columns for the word, its definition, and extended information (illustrated below):

Complete script:

from readmdict import MDX
import re, json, csv

def get_describe(describe):
    if isinstance(describe, (list, tuple)):
        return ';'.join(get_describe(i) for i in describe)
    else:
        return describe

def deal_node(node, result=[], num=-1):
    chars = "■□◆▲●◇△○★☆"
    for k, (d, cs) in node.items():
        if num >= 0:
            d = d.replace('
', '')
            result.append(f"{'    '*num}{chars[num]} {k}: {d}")
        if cs:
            deal_node(cs, result, num+1)

def get_row(topic):
    id2children = {}
    root = {}
    for d in topic:
        node = id2children.get(d.get("parentid"), root)
        tmp = {}
        node[d['id']] = (get_describe(d['describe']), tmp)
        id2children[d['id']] = tmp
    name, (describe, _) = list(root.items())[0]
    txts = []
    deal_node(root, txts)
    other = "
".join(txts)
    return name, describe, other

mdx_file = "help.mdx"
mdx = MDX(mdx_file, encoding='utf-8')
items = mdx.items()
data = []
for key, value in items:
    word = key.decode().strip()
    topic = json.loads(re.findall('"data":(\[.+\])}\);', value.decode())[0])
    name, describe, other = get_row(topic)
    data.append((name, describe, other))

with open(mdx_file.replace('.mdx', '-UTF8 .csv'), 'w', newline='', encoding='u8') as f:
    cw = csv.writer(f, delimiter=',')
    cw.writerow(["Word", "Definition", "Extension"])
    cw.writerows(data)