Integrate Custom rJava Algorithms into Transwarp Discover for Scalable ML

Many machine learning engines such as Transwarp Discover include a large set of built‑in algorithms (statistics, classification, clustering, regression, association, neural networks). Users who already have their own algorithms may face compatibility issues when trying to use both custom and built‑in libraries.

To solve this, Discover 4.6 introduces a one‑stop rJava development feature that allows custom algorithms to run alongside the built‑in library. Users download the one‑stop engineering package, write their algorithm in Scala or Java, compile it into a JAR, and upload it through the Discover RStudio interface.

Step‑by‑step rJava development

Download the one‑stop project file rJavaApp.rar (link: https://pan.baidu.com/s/1boLaFRT, password: e45y).

Open the extracted project in IntelliJ IDEA, create a new Scala or Java script, and implement the algorithm.

Compile and package the code into a .jar file.

Upload the JAR to the Discover cluster via the “upload” function in RStudio.

Use the Discover R functions txAdd and txGrant to load the JAR into HDFS and optionally share it with other users.

K‑Means case study

The article demonstrates a custom K‑Means implementation that improves initial centroid selection based on data density, leading to higher clustering accuracy. After building the JAR, the algorithm is uploaded and invoked in RStudio. Visual comparisons show the traditional K‑Means results versus the new algorithm on Shanghai merchant location data.

Results indicate the custom K‑Means achieves slightly better clustering precision, illustrating that extending Discover with user‑developed algorithms is straightforward and enhances flexibility.

Overall, rJava autonomous development bridges R and Java/Scala, enabling data scientists familiar with Java/Scala to deploy their models on the Discover platform while leveraging its extensive built‑in algorithm library.