DBLE LOAD DATA: Design Overview and Source Code Walkthrough

The article introduces DBLE's implementation of the LOAD DATA command, which enables large‑scale data import from text files into tables, mirroring MySQL's protocol while adding middleware-specific handling for storage, routing, and backend interaction.

Design Overview

DBLE simulates MySQL's LOAD DATA processing by intercepting the command, handling client‑side data transfer, routing decisions, and forwarding the processed data to the appropriate MySQL backend nodes. The overall flow is illustrated with a diagram (omitted).

Key Classes

The functionality revolves around two main classes:

ServerLoadDataInfileHandler : Manages client‑side interaction, parses the LOAD DATA command, requests files from the client, and handles received data.

LoadDataUtil : Handles communication with the backend MySQL servers, sending the processed file data.

Source Code Walkthrough

When a client issues a LOAD DATA statement, DBLE receives it in ServerQueryHandler#query and dispatches to FrontendConnection#loadDataInfileStart :

public void query(String sql) {
    ServerConnection c = this.source;
    if (LOGGER.isDebugEnabled()) {
        LOGGER.debug(String.valueOf(c) + sql);
    }
    // ...
    int rs = ServerParse.parse(sql);
    int sqlType = rs & 0xff;
    switch (sqlType) {
        case ServerParse.LOAD_DATA_INFILE_SQL:
            c.loadDataInfileStart(sql);
            break;
        // ...
    }
}

The loadDataInfileStart method creates a ServerLoadDataInfileHandler and calls its start method:

public void loadDataInfileStart(String sql) {
    if (loadDataInfileHandler != null) {
        try {
            loadDataInfileHandler.start(sql);
        } catch (Exception e) {
            LOGGER.info("load data error", e);
            writeErrMessage(ErrorCode.ERR_HANDLE_DATA, e.getMessage());
        }
    } else {
        writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR, "load data infile sql is not unsupported!");
    }
}

The start method parses parameters, determines whether the file is local, and either requests the file from the client or reads a local file, then routes each line to the appropriate backend node:

public void start(String strSql) {
    // ...
    parseLoadDataPram();
    if (statement.isLocal()) {
        isStartLoadData = true;
        // request file from client
        ByteBuffer buffer = serverConnection.allocate();
        RequestFilePacket filePacket = new RequestFilePacket();
        filePacket.setFileName(fileName.getBytes());
        filePacket.setPacketId(1);
        filePacket.write(buffer, serverConnection, true);
    } else {
        if (!new File(fileName).exists()) {
            String msg = fileName + " is not found!";
            clear();
            serverConnection.writeErrMessage(ErrorCode.ER_FILE_NOT_FOUND, msg);
        } else {
            if (parseFileByLine(fileName, loadData.getCharset(), loadData.getLineTerminatedBy())) {
                RouteResultset rrs = buildResultSet(routeResultMap);
                if (rrs != null) {
                    flushDataToFile();
                    isStartLoadData = false;
                    serverConnection.getSession2().execute(rrs);
                }
            }
        }
    }
}

Incoming file data from the client is handled by ServerLoadDataInfileHandler#handle , which stores small files in memory or writes larger ones to disk:

public void handle(byte[] data) {
    try {
        if (sql == null) {
            clear();
            serverConnection.writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR, "Unknown command");
            return;
        }
        BinaryPacket packet = new BinaryPacket();
        ByteArrayInputStream inputStream = new ByteArrayInputStream(data, 0, data.length);
        packet.read(inputStream);
        // store the received data
        saveByteOrToFile(packet.getData(), false);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

After the client signals the end of transmission, ServerLoadDataInfileHandler#end finalizes routing, builds the result set, and dispatches LOAD DATA commands to backend MySQL nodes:

public void end(byte packId) {
    isStartLoadData = false;
    this.packID = packId;
    // empty packet for end
    saveByteOrToFile(null, true);
    if (isHasStoreToFile) {
        parseFileByLine(tempFile, loadData.getCharset(), loadData.getLineTerminatedBy());
    }
    // build route result set and execute on backends
    RouteResultset rrs = buildResultSet(routeResultMap);
    if (rrs != null) {
        flushDataToFile();
        serverConnection.getSession2().execute(rrs);
    }
}

Finally, LoadDataUtil#requestFileDataResponse sends the processed data to each backend MySQL connection, either directly from memory or by streaming a temporary file:

public static void requestFileDataResponse(byte[] data, BackendConnection conn) {
    byte packId = data[3];
    MySQLConnection c = (MySQLConnection) conn;
    RouteResultsetNode rrn = (RouteResultsetNode) conn.getAttachment();
    LoadData loadData = rrn.getLoadData();
    List
loadDataData = loadData.getData();
    if (loadDataData != null && loadDataData.size() > 0) {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        for (String line : loadDataData) {
            String s = line + loadData.getLineTerminatedBy();
            bos.write(s.getBytes(CharsetUtil.getJavaCharset(loadData.getCharset())));
        }
        packId = writeToBackConnection(packId, new ByteArrayInputStream(bos.toByteArray()), c);
    } else {
        BufferedInputStream in = new BufferedInputStream(new FileInputStream(loadData.getFileName()));
        packId = writeToBackConnection(packId, in, c);
    }
    // ...
}

In summary, the article provides a comprehensive analysis of DBLE's LOAD DATA implementation, covering the design, key classes, and step‑by‑step source code examination to help readers understand the complete data import workflow.