Tagged articles
UDF
29 articles · Page 1 of 1
This article details the comprehensive redesign of a mobile video‑editing page, covering background challenges, requirement analysis, MVVM + UDF architectural choices, module decomposition, dependency‑injection implementation, and the design of a robust Redo/Undo system to improve maintainability and user experience.
MVVMUDFarchitecture
0 likes · 19 min read

This article introduces RisingWave, an open‑source streaming database, and explains how its SQL‑based interface, compute‑storage separation, UDF support, and materialized views enable efficient real‑time feature engineering, state management, and diverse downstream applications, including the enhancements in RisingWave 2.0.
RisingWaveSQLState Management
0 likes · 17 min read

This comprehensive ODPS (MaxCompute) development guide serves as a mini‑encyclopedia, detailing common parameter tuning, built‑in SQL functions, step‑by‑step Java UDF creation, job lifecycle insights, and practical performance‑optimization techniques such as parallelism adjustment, map‑join hints, and small‑file mitigation.
MaxComputeODPSSQL
0 likes · 26 min read

This article introduces a unified User‑Defined Function (UDF) solution on a cloud data platform, detailing its remote execution architecture, compatibility with Hive UDFs, resource isolation, hot‑update capabilities, internal platform implementation, open‑source contributions to PrestoDB, and future work plans.
HiveUDFcloud platform
0 likes · 11 min read

This article explains two methods for keeping MySQL data in sync with a Redis cache—using MySQL triggers with a UDF function and parsing MySQL binlog streams—detailing their processes, advantages, limitations, and related open‑source tools like Canal.
CanalDatabase ReplicationMySQL
0 likes · 7 min read

This article explains two technical solutions for keeping MySQL data in sync with Redis cache—using a MySQL trigger with a UDF function and parsing MySQL binlog events—while also reviewing the Canal open‑source tool and discussing their trade‑offs and implementation details.
CanalDatabase ReplicationMySQL
0 likes · 6 min read

This article compares two approaches for keeping MySQL and Redis in sync—using MySQL triggers combined with a UDF to write directly to Redis, and parsing MySQL binlogs (e.g., with Alibaba's Canal) to propagate changes—detailing their workflows, advantages, limitations, and implementation details.
CanalMySQLRedis
0 likes · 7 min read


ITPUBJul 1, 2022 · Databases
This article introduces Apache IoTDB, an open‑source time‑series database for industrial IoT, outlines its recent feature releases, explains its data‑modeling and compression strategies, and discusses UDF, trigger, and quality‑control capabilities that guide technical selection and architecture design.
Apache IoTDBData ModelingIndustrial IoT
0 likes · 12 min read

The study compares MaxCompute JSON extraction functions—FROM_JSON, get_json_object, and custom JMESPath/JSONPath UDFs—showing simple field extraction with get_json_object is fastest, while complex queries benefit from FROM_JSON or JMESPath, and outlines corresponding JSON generation methods and best‑practice recommendations.
DataEngineeringJMESPathJSON
0 likes · 11 min read

This article compiles a comprehensive list of common production problems encountered with Apache Flink, covering cluster sizing, checkpoint failures, backpressure analysis, resource allocation, deployment errors, UDF definitions, data skew, Kafka configurations, and provides detailed troubleshooting steps and best‑practice recommendations.
Apache FlinkKafkaUDF
0 likes · 39 min read

This article provides an in‑depth overview of Spark SQL, covering its architecture, DataSet/DataFrame creation, DSL and SQL usage, integration with Hive, custom UDF/UDAF/Aggregator implementations, handling of small files, Cartesian product detection, and a catalog of useful built‑in functions and window operations.
DatasetHiveSpark SQL
0 likes · 29 min read

This article provides a comprehensive guide to Flink Table and SQL window semantics—including group, tumbling, sliding, and session windows—covers over windows, demonstrates how to define windows in SQL, explains built‑in functions, shows how to implement scalar, table, aggregate and table‑aggregate UDFs, and details Flink's integration with Hive, complete with Maven dependencies and runnable examples.
FlinkHive IntegrationSQL
0 likes · 27 min read

This article provides a comprehensive overview of Spark SQL, covering its architecture, DataSet/DataFrame APIs, code examples for creating and querying datasets, join strategy selection, handling Hive tables, small‑file issues, inefficient NOT‑IN subqueries, Cartesian products, and a catalog of useful built‑in functions.
DatasetHive IntegrationSpark SQL
0 likes · 40 min read

This article explains two methods for keeping MySQL data in sync with a Redis cache—using MySQL triggers combined with a UDF function and parsing MySQL binlog via Alibaba's Canal—detailing their principles, implementation steps, advantages, limitations, and practical deployment considerations.
CanalMySQLUDF
0 likes · 6 min read

The article explains two technical methods for keeping MySQL and Redis in sync—using MySQL triggers with a custom UDF to write directly to Redis, and parsing MySQL binlog streams (or using Alibaba's Canal) to propagate changes, while discussing their suitable scenarios, challenges, and implementation details.
CanalDatabase ReplicationMySQL
0 likes · 5 min read

This article explains two technical methods for keeping MySQL data in sync with a Redis cache—using MySQL triggers combined with a UDF function and parsing MySQL binary logs (with Canal)—detailing their workflows, advantages, limitations, and implementation considerations.
CanalDatabase ReplicationRedis
0 likes · 6 min read

This guide walks through setting up Apache Zeppelin as a low‑cost, SQL‑centric development platform for Flink, covering environment preparation, installation, interpreter configuration, execution modes, verification, common pitfalls, dimension‑table joins, custom UDFs, Redis integration, and dual‑stream join techniques.
FlinkRedisSQL
0 likes · 24 min read

DiDi migrated over 10,000 Hive SQL tasks to Spark SQL using a lightweight dual‑run pipeline that extracts, rewrites, compares, and switches tasks, fixing syntax and UDF differences while adding features such as small‑file merging and enhanced partition pruning, resulting in Spark handling 85 % of workloads with 40 % faster execution, 21 % lower CPU and 49 % lower memory usage.
DataMigrationHiveSQLOptimization
0 likes · 18 min read

To dramatically improve Hive UDF performance when converting coordinates to administrative districts, this article compares two implementation strategies, details the technical challenges of repeatedly loading a 157 MB Geo data file, and presents a static‑cached solution that reduces query time from seconds to milliseconds, achieving roughly a 140‑fold speed increase.
HiveStatic CachingUDF
0 likes · 15 min read

This article demonstrates how to create and register custom scalar UDFs, untyped and type‑safe aggregate functions (UDAF and Aggregator) in Spark SQL, and how to apply window functions such as ROW_NUMBER, providing complete Scala code examples and execution results.
AggregatorSQLScala
0 likes · 16 min read

This tutorial explains several techniques for adding new columns to PySpark DataFrames—including native Spark functions, user‑defined functions, RDD transformations, Pandas UDFs, and SQL queries—while demonstrating data loading, schema handling, and code examples for each method.
Column CreationPySparkSQL
0 likes · 9 min read

The article reviews Google’s F1 Query paper, describing its architecture, three execution modes, data source handling, extensibility features such as UDF servers and TVFs, and performance optimizations that enable a unified, enterprise‑wide SQL engine for heterogeneous big‑data workloads.
Data PartitioningF1 QuerySQL
0 likes · 23 min read

This article explains PySpark's multi‑process architecture, how the Python driver uses Py4J to call Java/Scala APIs, the implementation of RDD and DataFrame interfaces, executor‑side process communication and serialization with Arrow, and the design of Pandas UDFs, while also discussing current limitations and future directions.
ArrowPySparkPython
0 likes · 13 min read

This article explains how to register scalar UDFs, table‑valued UDTFs, and aggregate UDAFs in Apache Flink, illustrates typical compilation and runtime pitfalls with concrete Scala code examples, and provides corrected implementations and best‑practice tips for reliable function registration.
Apache FlinkScalaUDAF
0 likes · 13 min read

This article introduces Spark SQL fundamentals, including its architecture, DataFrame and Dataset abstractions, query methods, interoperability with RDD, user-defined functions, integration with Hive, data source handling, and provides step‑by‑step Scala code examples for loading data, performing aggregations, and solving common analytical tasks.
DataFramesHiveSQL
0 likes · 15 min read

The article explains MySQL Group Replication 8.0.16's new segmented communication protocol, version‑compatibility rules for adding members, and introduces two UDFs that allow administrators to set and query the protocol version to maintain high‑availability across mixed‑version clusters.
Communication ProtocolDatabase VersioningGroup Replication
0 likes · 6 min read

This article explores MySQL’s advanced features—including partition tables, views, stored procedures, triggers, foreign key constraints, bind variables, user‑defined functions, and character set considerations—explaining their principles, usage patterns, performance implications, and practical tips for large‑scale data scenarios.
MySQLPartitioningUDF
0 likes · 35 min read

This article shares a comprehensive set of practical tips and solutions for common Spark SQL issues—including out‑of‑memory errors, UDF‑induced GC, thread blocking, system‑property initialization, speculation side‑effects, accumulator traps, concurrent job scheduling, and excessive logging—helping engineers improve stability and efficiency of their Spark‑based financial systems.
AccumulatorPerformance TuningSpark
0 likes · 15 min read

This tutorial explains how to create, register, and deploy custom scalar, aggregation, and window functions for the Presto distributed query engine using Java annotations, the Presto plugin mechanism, and code examples that illustrate UDF development, plugin packaging, and state handling for aggregation functions.
JavaSQLUDF
0 likes · 11 min read
