Jkes: A Java‑Kafka‑ElasticSearch Search Framework – Installation, Configuration, and Usage Guide

Jkes is a Java‑based search framework that integrates Kafka and ElasticSearch, offering annotation‑driven JPA‑style object/document mapping and REST APIs for document indexing and retrieval.

Installation can be done by adding the jkes-index-connector and jkes-delete-connector to the Kafka Connect classpath and installing the Smart Chinese Analysis Plugin with sudo bin/elasticsearch-plugin install analysis-smartcn.

Configuration includes adding the jkes-spring-data-jpa dependency and enabling the framework with annotations such as @EnableAspectJAutoProxy, @EnableJkes, and defining a @Configuration class that provides a PlatformTransactionManager bean and a JkesProperties bean, e.g.:

@EnableAspectJAutoProxy
@EnableJkes
@Configuration
public class JkesConfig {
    @Bean
    public PlatformTransactionManager transactionManager(EntityManagerFactory factory, EventSupport eventSupport) {
        return new SearchPlatformTransactionManager(new JpaTransactionManager(factory), eventSupport);
    }
}

Additional beans such as JkesProperties can be defined by extending DefaultJkesPropertiesImpl and overriding methods to supply Kafka and ElasticSearch connection details.

Index API is demonstrated by annotating entity classes with @Document, @MultiFields, and related field annotations to automatically generate index mappings. Example entity definitions for Person and PersonGroup are provided.

@lombok.Data
@Entity
@Document
public class Person extends AuditedEntity {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    @MultiFields(
        mainField = @Field(type = FieldType.Text),
        otherFields = {
            @InnerField(suffix = "raw", type = FieldType.Keyword),
            @InnerField(suffix = "english", type = FieldType.Text, analyzer = "english")
        }
    )
    private String name;
    @Field(type = FieldType.Keyword)
    private String gender;
    @Field(type = FieldType.Integer)
    private Integer age;
    // other fields ...
}

The framework automatically indexes documents on entity save and removes them on delete, handling events via SaveEvent, DeleteEvent, and DeleteAllEvent.

Search API is exposed by the jkes-search-service Spring Boot application (default port 9000). Various query types are supported, including URI queries, nested queries, match, bool, source filtering, prefix, wildcard, and regexp queries, illustrated with curl examples.

curl -XPOST localhost:9000/api/v1/integration_test_person_group/person_group/_search?from=3&size=10

The working principle of indexing involves scanning @Document‑annotated entities at startup, generating index and mapping JSON, creating Kafka‑ElasticSearch connectors, and intercepting repository methods to emit events that are sent to Kafka and eventually persisted in ElasticSearch. Deletion events are handled similarly, with DeleteAllEvent directly invoking the ElasticSearch client to drop and recreate indices.

Query processing is performed by a separate Spring Boot service that receives REST requests, forwards them to ElasticSearch via the Java Rest Client, and returns processed results; this decoupling allows future integration of machine‑learning‑based ranking.

Module Overview :

jkes-core – core annotations, ElasticSearch and Kafka integration, metadata handling, event model, utilities.

jkes-boot – integration with third‑party frameworks (currently Spring Data JPA), providing AOP‑based event interception and transaction management.

jkes-services – includes jkes-delete-connector (Kafka connector for deletions) and jkes-search-service (RESTful search API).

jkes-integration-test – a Spring Boot test project for functional verification and throughput measurement.

Development requires a recent Kafka version and can be built with Maven. The project is open‑source on GitHub, licensed under Apache License 2.0, and welcomes contributions.