Improving Product Quality through Code Vulnerability Inspection and Deep Code‑Search Techniques

Background and Motivation – Product failures are often caused by code defects; examples include banking software breaches, rocket launch failures, and large‑scale power outages. Early detection of code vulnerabilities can reduce 70‑80% of crashes and security incidents, making thorough code inspection essential for high‑quality products.

When to Perform Vulnerability Checks – The later a defect is found in the release pipeline, the higher the remediation cost. Testing phases offer the lowest cost, so all code‑level flaws should be identified before product launch.

How to Perform Checks

1. Source‑code scanning – Enforces coding standards across four categories (Error, Security, Forbidden, Recommendation). Custom rules can be defined per business scenario to catch violations.

2. Binary scanning – Tools such as Google’s Veridex detect illegal API calls and classify them into three risk levels.

Deep Code‑Search Techniques

Challenges – Determining code features, slow search speed, sparse information, massive data volume (tens of millions of files), and slow ingestion.

Architecture – A five‑layer system: Python backend for incremental updates, MySQL for primary storage, Sphinx for real‑time distributed indexing, PHP+NGINX for API services, and a front‑end UI for result display.

Incremental Ingestion Pipeline – Steps include retrieving repository URLs, extracting commit dates, parsing logs, deduplication, downloading files, tokenizing, and updating the index. Example SVN commands:

svn log -r {0} --xml -v "{1}" --username "{2}" --password "{3}" --non-interactive --no-auth-cache --trust-server-cert > {4}
svn export -r {0} "{1}" "{2}" --force --username {3} --password "{4}" --non-interactive --no-auth-cache --trust-server-cert

Deduplication – For SVN, use module_id + revision ; for Git, use repo_id + commit_sha1 . Example duplicate paths:

http://svn.example.com/svn/testxxx/111/222/333
http://svn.example.com/svn/testxxx/111

Git branch duplication is handled similarly.

Real‑time Distributed Indexing with Sphinx – Sphinx supports billions of documents and TB‑scale data. Commands:

/usr/local/sphinx/bin/indexer -c sphinx.conf code

/usr/local/sphinx/bin/searchd -c sphinx.conf &

/usr/local/sphinx/bin/search -c sphinx.conf mykeyword

Configuration example:

indexcoderealtime {
    type = rt
    path = /usr/local/sphinx/indexer/files/coderealtime
    rt_field = content
    rt_field = filename
    rt_attr_uint = rpid
    rt_attr_timestamp = cdate
}
indexcodedistributed {
    type = distributed
    local = coderealtime
    agent = localhost:9312:crt1
    agent = localhost:9312:crt2
}
searchd {
    listen = 9312
    listen = 9306:mysql41
    log = /usr/local/sphinx/indexer/logs/searchd.log
    query_log = /usr/local/sphinx/indexer/logs/query.log
}

Ranking Method – Results are ordered by phrase score, commit time, and BM25 algorithm. BM25 formula:

Methods to Improve Product Quality

1. Combine business oversight with vulnerability remediation – after scanning, deep‑search hidden bugs across the entire codebase and coordinate with owners to fix them.

2. Sensitive‑word and forbidden‑API checks in audit systems and signature verification.

A demo UI shows search input, filters (time, language, owner, repository) and results (file name, repo, path, version, date, owner), enabling rapid identification of responsible developers.

Conclusion and Outlook

The presented three‑part approach—background/methods, deep code‑search technology, and quality‑improvement tactics—demonstrates how code‑search accelerates defect localization, improves ranking, and boosts overall software quality. Future work will explore code recommendation using semantic context and AI, as well as function‑level recommendation.