Build a High‑Performance Go File Transfer System with Custom Protocol and Zero‑Copy

Project Structure

fs_transfer_project/
├── proto/                # Protobuf message definitions
├── frame/                # Custom frame protocol read/write
│   ├── frame.go
│   └── command.go       # CMD enum definitions
├── server/               # Server implementation
│   ├── main.go          # TCP listener & connection dispatch
│   ├── dispatcher.go    # CMD dispatch logic
│   ├── file_ops.go      # Basic file operations
│   └── file_ops_advanced.go # Zero‑copy, resume, CRC32
└── client/               # Client implementation
    ├── main.go          # Client entry point
    ├── download_demo.go # File download flow
    └── upload_demo.go   # File upload flow

Core Design

The server communicates using a custom binary frame that carries a Protobuf‑encoded command payload. Each frame consists of a fixed‑size header followed by a variable‑size payload, enabling multiplexed and extensible interactions.

Frame Header Layout

Magic (uint32): frame identifier

Version (byte): protocol version

Flags (uint8): flag bits

Cmd (uint16): command type

StreamID (uint32): multiplexing identifier

PayloadLen (uint32): length of the payload

Checksum (uint16): optional integrity verification

Reading a Frame

func ReadFrame(r *bufio.Reader) (Header, []byte, error) {
    var h Header
    hdr := make([]byte, HeaderSize)
    if _, err := io.ReadFull(r, hdr); err != nil {
        return h, nil, err
    }
    h.Magic = binary.BigEndian.Uint32(hdr[0:4])
    if h.Magic != MagicValue {
        return h, nil, errors.New("invalid magic")
    }
    h.Version = hdr[4]
    h.Flags = hdr[5]
    h.Cmd = binary.BigEndian.Uint16(hdr[6:8])
    h.StreamID = binary.BigEndian.Uint32(hdr[8:12])
    h.PayloadLen = binary.BigEndian.Uint32(hdr[12:16])
    payload := make([]byte, h.PayloadLen)
    if _, err := io.ReadFull(r, payload); err != nil {
        return h, nil, err
    }
    // optional checksum verification
    expectedChecksum := binary.BigEndian.Uint16(hdr[4:6])
    if expectedChecksum != 0 {
        actualChecksum := calculateChecksum(payload)
        if actualChecksum != expectedChecksum {
            return h, nil, errors.New("checksum mismatch")
        }
    }
    return h, payload, nil
}

The Cmd field is used by the server dispatcher to route the request to the appropriate handler (e.g., CmdAuth, CmdOpen, CmdRead, CmdWrite).

Advanced File Operations

Zero‑Copy File Sending

On Linux the sendfile syscall transfers data directly between file descriptors, eliminating user‑space copies. On non‑Linux platforms the implementation falls back to chunked read/write.

func sendFileZeroCopy(conn net.Conn, path string) error {
    if runtime.GOOS != "linux" {
        return fmt.Errorf("sendfile zero-copy only supported on Linux")
    }
    f, err := os.Open(path)
    if err != nil { return err }
    defer f.Close()
    tcpConn, ok := conn.(*net.TCPConn)
    if !ok { return fmt.Errorf("connection is not TCP") }
    fileConn, err := tcpConn.File()
    if err != nil { return err }
    defer fileConn.Close()
    fi, _ := f.Stat()
    off := int64(0)
    size := fi.Size()
    for off < size {
        n, err := unix.Sendfile(int(fileConn.Fd()), int(f.Fd()), &off, int(size-off))
        if err != nil { return err }
        if n == 0 { break }
        off += int64(n)
    }
    return nil
}

Chunked Read/Write & Resume

Read and write fixed‑size blocks.

Record which blocks have been transferred.

On resume, only missing blocks are sent.

func writeChunk(f *os.File, offset int64, data []byte) error {
    _, err := f.WriteAt(data, offset)
    return err
}

Data Integrity with CRC32

func computeChecksum(data []byte) uint32 {
    return crc32.ChecksumIEEE(data)
}

Server Core Code

TCP Listener & Connection Dispatch

ln, _ := net.Listen("tcp", ":9000")
for {
    conn, _ := ln.Accept()
    go handleConn(conn)
}

Dispatcher Example

switch hdr.Cmd {
case frame.CmdAuth:
    var req fsproto.AuthReq
    if err := proto.Unmarshal(payload, &req); err != nil {
        logger.Error("Failed to unmarshal AuthReq", zap.Error(err))
        return
    }
    handleAuth(conn, req)
case frame.CmdList:
    var req fsproto.ListReq
    if err := proto.Unmarshal(payload, &req); err != nil {
        logger.Error("Failed to unmarshal ListReq", zap.Error(err))
        return
    }
    handleList(conn, req)
case frame.CmdOpen:
    var req fsproto.OpenReq
    if err := proto.Unmarshal(payload, &req); err != nil {
        logger.Error("Failed to unmarshal OpenReq", zap.Error(err))
        return
    }
    handleOpen(conn, req)
case frame.CmdRead:
    var req fsproto.ReadReq
    if err := proto.Unmarshal(payload, &req); err != nil {
        logger.Error("Failed to unmarshal ReadReq", zap.Error(err))
        return
    }
    handleRead(conn, req)
case frame.CmdWrite:
    var req fsproto.WriteReq
    if err := proto.Unmarshal(payload, &req); err != nil {
        logger.Error("Failed to unmarshal WriteReq", zap.Error(err))
        return
    }
    handleWrite(conn, req)
default:
    logger.Warn("Unknown command", zap.Uint16("cmd", hdr.Cmd))
}

Client Core Code

Authentication Example

authReq := &fsproto.AuthReq{Token: "secret_token", ClientId: "client_001"}
payload, _ := proto.Marshal(authReq)
frame.WriteFrame(conn, frame.Header{Version: 1, Cmd: frame.CmdAuth}, payload)

File Download Example

frame.WriteFrame(conn, frame.Header{Cmd: frame.CmdOpen}, openReqBytes)
for offset := int64(0); !eof; offset += chunkSize {
    readReq := &fsproto.ReadReq{Handle: handle, Offset: offset, Length: chunkSize}
    payload, _ := proto.Marshal(readReq)
    frame.WriteFrame(conn, frame.Header{Cmd: frame.CmdRead}, payload)
    // process received chunk
}

Performance Optimizations & Future Extensions

Multiplexing : multiple files share a single TCP connection.

Zero‑Copy : Linux sendfile for large‑file transmission.

Chunked Checksums : CRC32 per block ensures integrity.

Protobuf Command Extension : new CMD values can be added without breaking compatibility.

Potential extensions include TLS encryption, parallel connections, and Linux epoll for higher concurrency.

Source Code

GitHub: https://github.com/louis-xie-programmer/fs_transfer_project

Gitee mirror: https://gitee.com/louis_xie/fs_transfer_project