Build Real-Time Data Lake Analytics with Flink, Paimon, and EMR Serverless Spark

Serverless Spark Job Scheduling

Serverless Spark also supports job scheduling. After publishing a developed task, you can create a workflow, orchestrate tasks, and configure a scheduling policy to run periodic jobs, such as Compact for Paimon tables.

In the task development page, write and publish the Paimon Compact SQL.

CALL paimon.sys.compact (</code><code>  table => 'test_paimon_db.test_append_tbl',</code><code>  partitions => 'dt="2024-06-04",hh="12"',</code><code>  order_strategy => 'zorder',</code><code>  order_by => 'category'</code><code>);

In the workflow page, create a workflow, add a node, and bind it to the published task.

Each workflow node can specify its own engine version and Spark configuration, such as:

spark.sql.extensions                org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions</code><code>spark.sql.catalog.paimon            org.apache.paimon.spark.SparkCatalog</code><code>spark.sql.catalog.paimon.metastore  dlf</code><code>spark.sql.catalog.paimon.warehouse  oss://test/warehouse

Manually run the workflow.

After successful execution, verify the Compact effect by querying the Paimon $files system table.

SELECT file_path, record_count, file_size_in_bytes FROM `paimon`.`test_paimon_db`.`test_append_tbl$files` WHERE partition='[2024-06-04, 12]';

Result before Compact:

Result after Compact: