This article explains how to extend a Flink CDC job that already syncs an entire MySQL database to Doris so that newly created tables are automatically created in Doris in real time, using the CdcTools utility, side‑output streams, and asynchronous I/O.
This article explains how to use Flink CDC together with the CdcTools utility to automatically capture newly created MySQL tables and synchronize both their schema and data to a Doris database in real time, covering the required code, side‑output handling, async execution, and a special delete‑sign field.
This article explains how to inject the LanceSpark plugin into Spark, covering the core LanceSparkSessionExtensions class, various ways to register extensions, the custom parser and planner strategy implementations, and the underlying Spark mechanisms such as injectParser, injectPlannerStrategy, and PredicateHelper.
The article examines the recent push for modern features in Java, contrasts them with Scala implementations that have existed for years, and walks through functional programming, pattern matching, immutable collections, type inference, string interpolation, sealed classes, and concurrency with side‑by‑side code examples in both languages.
This article explains Spark SQL’s Catalyst optimizer, describing its extensible design, tree‑based representation, rule‑driven transformations, batch execution to a fixed point, and how Scala’s pattern matching and quasiquotes enable efficient analysis, logical optimization, physical planning, and code generation.
This article describes how Tubi improved the latency of its Redis‑backed online feature store for machine‑learning inference by analyzing query patterns, measuring client‑side bottlenecks, and applying optimizations such as binary Avro encoding, MGET usage, virtual partitioning, and parallel deserialization to meet a sub‑10 ms SLA.
The article compares Java, Scala, SQL, and Python for data‑engineering tasks, arguing that Python’s versatility, rich ecosystem, and ease of use make it the preferred language for both small‑scale and massive Spark workloads despite its performance trade‑offs.
This talk, the third in a Spark series, introduces the Spark SQL parsing layer, explains its architecture and integration with ANTLR4, details core implementation classes, and presents a real‑world optimization case that reduces code complexity and improves maintainability.
This guide analyzes common Spark Out‑Of‑Memory scenarios—such as massive data volumes, data skew, and improper resource allocation—and provides step‑by‑step configurations, memory‑management tweaks, partitioning strategies, and shuffle optimizations to prevent OOM failures in production.
This article explains the sidecar concept, compares it with SDK approaches, and provides detailed Kubernetes examples—including a log‑collection sidecar, a request‑forwarding sidecar, and an HTTP‑intercepting sidecar—complete with YAML manifests and Rust and Scala code to demonstrate implementation and deployment.
The article reviews a Scala meetup where experts compare Java and Scala, highlighting Scala's stronger expressiveness, type inference, pattern matching, safety, and concurrency features, and discusses real‑world adoption, developer experiences, and a recruitment opportunity for a Scala‑focused big‑data team.
This article reviews Alibaba's Tongyi Qianwen large language model, shares personal impressions, provides a fish‑flavored pork recipe, conducts a SWOT analysis, and includes Scala Spark and Java code examples illustrating its capabilities and usage scenarios.
The article reviews Alibaba’s Tongyi Qianwen large‑language model, shares a cooking recipe generated by the AI, presents a SWOT analysis, and provides code examples—including a Spark Scala script for same‑period month‑over‑month calculations and a Java bubble‑sort implementation.
IntelliJ IDEA 2023.1 introduces a refreshed UI, faster Maven imports, background commit checks, enhanced Java 20 support, improved Scala tooling, new web framework integrations, and numerous editor and user‑experience enhancements, delivering a more modern and efficient development environment.
IntelliJ IDEA 2023.1 introduces a revamped UI with new split options, compact mode, and redesigned widgets, boosts performance through faster Maven imports and smarter indexing, adds background commit checks, enhances Java, Scala, and web development support, and improves overall user experience with scaling and AI‑powered search.
The eighth Scala Meetup featured three technical talks—Tubi’s journey with gRPC and Akka‑gRPC migration, an introduction to functional programming concepts in Scala, and the use of Scala in interactive notebooks—followed by audience Q&A and links to full recordings and slides.
This article explains Spark SQL's CacheManager, how it stores cached query results using InMemoryRelation, the ways to trigger and release caches, the internal data structures like IndexedSeq and CachedData, and the role of canonicalization in determining cache reuse.
This guide explains how to download JDK, Maven, Scala and Spark 3.3.0, modify the Spark pom and configuration files for CDH 6.3.2, compile Spark with Maven, deploy the binaries to a client node, set up spark‑sql and spark‑submit scripts, and address common runtime issues.
The seventh Scala Meetup gathered over 1400 online participants to share three technical talks covering Tubi's content recommendation system architecture, philosophical insights into the nature of computation, and practical experiences with reactive streams in large‑scale JVM environments, followed by a round‑table discussion and audience feedback.
The article introduces BIDMach, a compact Scala‑based machine‑learning framework built with JNI‑driven CUDA/MKL, explains its three‑layer architecture, and discusses broader considerations for designing usable, high‑performance, and extensible industrial AI frameworks, emphasizing co‑design, algorithm‑framework co‑evolution, and ecosystem factors.
The article explains how Spark, developed by UC Berkeley's AMP Lab, quickly surpassed MapReduce by offering faster execution, a simpler Scala‑based programming model, lazy RDD transformations, a rich ecosystem including SQL, Streaming, MLlib and GraphX, and practical code examples such as a three‑line WordCount.
This article guides readers through setting up a Kafka 2.7.0 source‑code environment, presents a comprehensive overview of Kafka's core modules, and outlines a step‑by‑step roadmap for deep‑dive analysis of both producer and broker implementations.
This article explains the origins of Spark DataFrames, compares them with RDDs, describes how Spark SQL optimizes DataFrame execution, and provides detailed examples of creating DataFrames from RDDs, files, and JDBC sources along with common DataFrame operations and code snippets.
This article provides a detailed introduction to FlinkSQL, covering its background, the Table API, required dependencies, differences between old and Blink planners, various API usage patterns, connector configurations for CSV, Kafka, Elasticsearch, MySQL, and how to convert between DataStream and Table in Flink's unified batch‑stream processing model.
This article provides a comprehensive guide to Apache Flink's Table API and SQL, covering required dependencies, the differences between old and Blink planners, program structure, table environment creation, catalog registration, query execution, conversion between DataStream and Table, update modes, and time attribute handling, with Scala code examples throughout.
This article explains the complete Kafka topic deletion workflow, from the client’s deleteTopics request through Zookeeper node creation, controller coordination, broker StopReplica handling, log renaming, delayed file removal, and final cleanup, while providing code excerpts and practical Q&A for common pitfalls.
This article introduces Mockito, a Java mocking framework, explains when to use it for testing code with external dependencies, and provides step‑by‑step guidance—including Maven setup, creating mock objects in Scala, and configuring mock behavior—complete with code snippets and reference links.
This article analyzes why Spark tasks fail with a "Task not serializable" exception when closures reference class members, demonstrates the issue with Scala code examples, and provides practical solutions such as using @transient annotations, moving functions to objects, and ensuring proper class serialization.
This article explains how to create a custom SparkListener in Apache Spark, provides Scala code examples for the listener and a main application, and details two registration approaches—via Spark configuration or SparkContext—along with a comprehensive list of listener event methods.
This article explains a Scala‑based DSL that uses macros and implicit parameters to generate type‑safe, boiler‑plate‑free conversion code between protobuf‑java messages and Scala case classes, showing examples, builder customisation, and upcoming Scala 3 support.
This article shares practical experience of building a high‑performance distributed prediction service by combining TensorFlow for Java with Spark‑Scala, covering framework selection, performance comparison, model training, loading, inference, deployment, and optimization techniques for large‑scale data processing.
This article explains how Spark transforms a logical execution graph into a physical job by partitioning stages, applying pipeline concepts, and generating tasks, while illustrating the process with detailed code examples and the internal workflow of job submission.
This article explains advanced Spark performance tuning techniques, focusing on diagnosing and resolving data skew and shuffle bottlenecks through stage analysis, key distribution inspection, and a variety of practical solutions such as Hive pre‑processing, key filtering, parallelism increase, two‑stage aggregation, map‑join, and combined strategies, while also covering ShuffleManager internals and related configuration parameters.
This article explains how to use Alibaba's Canal to capture MySQL binlog changes, forward them to Kafka, and process the incremental data in real time with Spark Streaming, including installation, configuration, client development, Spark code, testing, and troubleshooting dependency conflicts.
Tubi’s Popper platform combines a Scala‑based experiment engine, reproducible JSON‑stored configurations, a React UI, and data pipelines using Spark and Akka to enable fast, cross‑team A/B testing, automated analysis, health checks, and data‑driven decision making across mobile and OTT services.
This article examines why many developers favor Spark's mapPartitions over map, analyzes the underlying source code, highlights common pitfalls such as complexity and OOM risks, and provides practical guidelines and code examples for correctly using mapPartitions in both simple and advanced scenarios.
At Tubi, the Popper experiment engine—a Scala‑based, Akka‑powered backend service—combined with a self‑serve UI, automated analysis pipelines, and rigorous health checks, enables teams across ML, mobile, and OTT to run scalable A/B tests, rapidly iterate, and make data‑driven product decisions.
The 2021 Online Scala Meetup organized by Tubi featured four technical talks covering a fully reactive ad‑serving platform built with Scala and Akka‑Streams, a pragmatic take on functional programming, Scala 3 typeclass implementation, and a Spring‑Akka microservice integration, followed by summaries and recruitment information.
This guide demonstrates how to build a Docker‑based Spark standalone environment, configure Apache Zeppelin to connect to it, and perform data analysis on local CSV files, HDFS, and streaming sources such as Twitter and Kafka, with complete code examples.
This guide explains what Cerebro is, how to install it (including binary and Docker options), how to run it on Linux, macOS, and Windows, and how to use its UI to connect to an Elasticsearch node, view cluster overviews, manage shards, and execute DSL queries.
This article explains how Tubi customizes Spark SQL using lightweight macro‑based extensions to simplify column exclusion, JSON path queries, and other complex operations without modifying Spark's source code, detailing the two‑stage processing, example macros, and benefits for big‑data workloads.
Kafka handles many requests that require asynchronous processing or waiting for conditions by attaching a timeout parameter; if the condition isn’t met within the timeout, Kafka returns a timeout response, and it implements this efficiently using a hierarchical Timing Wheel data structure that offers O(1) insertion and fast expiration checks.
The article explains how Kafka handles timeout‑based requests using a Timing Wheel data structure, detailing its design, parameters, operation principles, overflow handling, and providing Scala code examples that illustrate O(1) task insertion compared to traditional O(logN) delay queues.
Delta Lake adds ACID transaction support, schema enforcement, data versioning, and unified batch‑and‑stream processing to Apache Spark‑based data lakes, addressing reliability, quality, performance, and update challenges of traditional data lake architectures.
The article summarizes a 2020 Scala online meetup, covering Akka's distributed state‑management for ad‑serving, Parsec parser combinators for arithmetic expression evaluation, AST manipulation with ANTLR, and an introduction to the functional ZIO library, while reflecting on personal takeaways.
This article provides a comprehensive guide on integrating Hive with Spark SQL, covering Hive‑on‑Spark and Spark‑on‑Hive setups, spark‑shell and spark‑sql usage, HiveServer2 with Beeline, Scala scripts for reading and writing Hive tables, and partition handling for aggregated results.
This article demonstrates how to use Flink 1.11's enhanced Hive integration to stream data from a Kafka source, write it into partitioned Hive tables with checkpoint‑driven commits, and read Hive tables as a continuous source using dynamic table options and table hints.
This article introduces Apache Hudi, explaining its storage types, query views, timeline feature, typical use cases such as near‑real‑time ingestion and incremental pipelines, and provides a step‑by‑step Scala/Spark quick‑start guide with code examples for compiling, inserting, updating, querying, and syncing data to Hive.
This article demonstrates how to implement an advertising business data statistics pipeline using Spark and Kudu, detailing metric requirements, Scala processing code, complex SQL aggregations, schema design, and data sinking for verification.
This article walks through the complete implementation of an advertising statistics pipeline using Spark and Kudu, covering requirement analysis, Scala code development, SQL queries, schema definition, and data sinking, with full code snippets and execution results.
This article walks through a Spark and Kudu advertising project, explaining the refactoring approach, Scala trait usage, implementation of ETL and province‑city statistics processors, and shows the complete Spark application entry point with full code examples.
This tutorial walks through a Spark‑Kudu advertising project that computes province‑city distribution statistics using SQL, defines the necessary schema, and demonstrates how to write the aggregated results back to a Kudu table for persistent storage, complete with Scala code examples.
This tutorial describes how to place advertising JSON data on HDFS, use Spark for ETL and analysis, enrich logs with IP lookup, and persist the results into Kudu with daily scheduling, including code examples and schema definitions.
This article introduces Complex Event Processing (CEP), explains its core concepts and features, details Flink's Pattern API with individual, combined, and group patterns, and provides a complete Scala example that detects three consecutive login failures within three seconds using Flink CEP.
Cat Talks is Tubi’s internal bi‑weekly series where engineers discuss foundational topics such as category theory, functors, natural transformations, free monads, parser combinators, and lenses, using live coding in Haskell and Scala to deepen understanding of functional programming concepts.
This article demonstrates how to implement real-time hot item ranking, page view counting, and unique visitor estimation using Apache Flink with Kafka sources, sliding windows, custom aggregation functions, and a Bloom filter backed by Redis, providing complete Scala code examples.
This article shares practical experiences and insights on using different types of state in Apache Flink for real‑time stream processing, covering managed versus raw state, code examples in Scala and Java, handling late data, dimension table joins, distinct semantics, and best‑practice recommendations.
This article explains how to use Spark Streaming's Direct Approach with Kafka, manually manage offsets, and provides complete Java and Scala implementations—including a JavaKafkaManager class, a demo application, and a Scala KafkaManager—illustrating the creation of DirectKafkaInputDStream, offset handling, and integration with Spark.
This article demonstrates how to create and register custom scalar UDFs, untyped and type‑safe aggregate functions (UDAF and Aggregator) in Spark SQL, and how to apply window functions such as ROW_NUMBER, providing complete Scala code examples and execution results.
This article introduces Apache Hudi’s core concepts and demonstrates how to operate in Copy‑On‑Write mode on a Spark‑based data lake, covering prerequisites, table types, configuration properties, upsert, incremental queries, and record deletion with Scala code examples.
This article explains the three time notions supported by Apache Flink—ProcessTime, EventTime, and IngestionTime—detailing their semantics, how Watermarks enable event‑time processing, and provides Scala code samples for configuring time characteristics, assigning timestamps, and generating Watermarks in a streaming job.
This article provides a detailed introduction to Spark Streaming, covering its architecture, DStream concepts, initialization, data sources, transformations, windowed aggregations, output operations, checkpointing, fault‑tolerance semantics, deployment, performance tuning, and monitoring for building reliable high‑throughput streaming applications.
This article provides a comprehensive overview of SparkSQL's logical plan architecture, detailing the stages of logical plan creation, analysis, rule‑based optimization, and the underlying catalog and rule systems that transform SQL queries into efficient execution plans.
This article examines how to run several Spark jobs concurrently on a shared SparkContext, balancing full CPU‑vcore utilization with the need to generate fewer Parquet files, and presents practical experiments, scheduling strategies, and performance results.
Aloha is a Scala‑implemented distributed task scheduling and management framework built on Spark that provides plug‑in extensions, high‑availability Master‑Worker architecture, custom event listeners, and a lightweight Scala‑based RPC layer for managing long‑running jobs such as Spark, Flink, and ETL tasks.
This article presents a series of hands‑on big‑data exercises, including Spark Scala data analysis, Kafka topic creation and custom partitioning, and MySQL table design with Scala‑based streaming calculations, providing complete source code and step‑by‑step solutions for each task.
On April 18, the inaugural 2020 Online Scala Meetup organized by Tubi featured four speakers who explored Scala design patterns, demonstrated ZIO STM usage, shared best practices for integrating Scala into Java codebases, and introduced Shapeless’s type-level programming, offering valuable insights for developers.
This article explains the differences between Spark on Hive and Hive on Spark, then provides step‑by‑step instructions for configuring Hive MetaStore, setting up SparkSQL to use Hive, and demonstrates a complete Scala program that creates a Hive table, loads data, and queries it.
This article introduces Spark SQL window functions, explains the difference between aggregation and window functions, and demonstrates how to use various ranking functions such as ROW_NUMBER, RANK, DENSE_RANK, and NTILE with practical Scala code examples and partitioning options.
This article explains how Apache Flink uses event‑time windows, watermarks, and allowed lateness to handle out‑of‑order and late data, and provides complete Scala code examples that demonstrate timestamp assignment, watermark generation, window triggering, and side‑output of late records.
This article provides an in‑depth, step‑by‑step guide to optimizing Spark jobs, covering development‑time best practices, resource‑parameter tuning, data‑skew detection and mitigation techniques, and shuffle‑stage performance tweaks, complete with Scala code examples and practical recommendations.
This article demonstrates how to generate 500 million visitor IDs with Spark, use map‑reduce operations to count occurrences, and identify the ID with the highest visit count, while discussing performance considerations such as memory spilling and cluster resources.
This article presents a step‑by‑step guide for building a real‑time data pipeline using Kafka as a message buffer, Spark‑Streaming's Direct Approach for processing, and HBase for storage, including code examples, Maven configuration, local cluster setup, and troubleshooting tips.
This article explains how to overcome Spark SQL's inability to handle certain Oracle data types, such as Timestamp with local timezone and FLOAT(126), by creating and registering a custom JdbcDialect that remaps unsupported types to compatible Spark types.
The November 3 Scala Meetup featured senior expert Deng Caoyuan’s whiteboard brainstorming and Tencent engineer Qu Guodong’s Cats tutorial, followed by a deep dive into Akka with real‑world examples such as a financial computing platform, real‑time article recommendation, web crawling, streaming, and practical tips for Scala developers.
This article explains why Tubi rewrote its legacy PHP ad platform, how it adopted Scala, Akka, and Reactive Streams to model the ad request lifecycle as a reactive stream, and details the first three processing steps—parsing, enrichment, and precise filtering—along with sample Scala filter code.
This article explains how to use Flink 1.9's built‑in Kafka connector, covering supported versions, Maven dependencies, consumer and producer configuration in Java and Scala, checkpointing, offset handling, partition discovery, timestamps, watermarks, and provides a complete runnable example.
This article explains two approaches—reflection‑based schema inference and programmatic schema definition—to transform a Spark RDD into a DataSet or DataFrame, demonstrates the required code, and discusses common Task‑not‑serializable errors with practical solutions.
This article explains the origins and motivations of Delta Lake, details its ACID transaction support, schema enforcement, metadata handling, versioning, and unified batch‑and‑stream processing, and provides a step‑by‑step Maven and Spark code tutorial for creating, updating, and querying Delta tables.
This article introduces the basic concepts of collaborative filtering, explains user‑based and item‑based approaches, presents co‑occurrence, Euclidean, Pearson, and Cosine similarity formulas, and provides complete Scala implementations for these metrics and association‑rule mining on the Spark platform, along with practical scalability tips.
This article explains how Spark Streaming's DStream abstraction is built from InputDStream through successive transform operators, details the internal ForEachDStream implementation, describes the job generation and scheduling workflow, and outlines how Beike's real‑time platform leverages these mechanisms for large‑scale streaming tasks.
This article provides a comprehensive overview of Apache Druid, covering its real‑time OLAP design, core features, six‑component architecture, segment storage model, data ingestion pipelines (including Tranquility and Kafka), native and SQL query interfaces, and practical tuning tips with code examples.
The article surveys functional programming fundamentals—contrasting it with imperative and declarative styles—and illustrates key concepts such as closures, currying, promises, partial functions, map/reduce, and divmod through Java, JavaScript, and Python examples, before highlighting Scala’s hybrid approach and the advantages of FP for writing elegant, maintainable, concurrent code.
This article walks through the SparkEnv initialization process in Apache Spark, detailing how the driver and executor environments are created, the key components such as SecurityManager, RpcEnv, SerializerManager, BroadcastManager, MapOutputTracker, ShuffleManager, MemoryManager, BlockManager, MetricsSystem, and OutputCommitCoordinator are instantiated, and how the final SparkEnv instance is assembled and stored.
This article explains how to register scalar UDFs, table‑valued UDTFs, and aggregate UDAFs in Apache Flink, illustrates typical compilation and runtime pitfalls with concrete Scala code examples, and provides corrected implementations and best‑practice tips for reliable function registration.
This article explains how to persist Spark DataFrames into Hive tables and specific partitions, covering the relevant write APIs, the need to select a database, and providing step‑by‑step Scala code examples for both Spark 1.6 and Spark 2.x versions, along with Hive table creation syntax.
Gatling‑Dubbo 2.0 is a Gatling‑based load‑testing plugin that replaces generic Dubbo invocations with real API calls, offering richer scenario orchestration, traffic models, native throttling, lower resource use, and higher concurrency, while providing Action, Check, and DSL components illustrated through a complete mixed‑scenario simulation.
This article explains the three time concepts in Flink—Process Time, Event Time, and Ingestion Time—illustrates their impact on windowed computations with examples, introduces watermarks and allowed lateness for handling out‑of‑order data, and provides complete Scala code for both processing‑time and event‑time streaming applications.
This article introduces Spark SQL fundamentals, including its architecture, DataFrame and Dataset abstractions, query methods, interoperability with RDD, user-defined functions, integration with Hive, data source handling, and provides step‑by‑step Scala code examples for loading data, performing aggregations, and solving common analytical tasks.
This article provides a detailed introduction to FlinkCEP, covering how to add the library, define simple and composite patterns, use quantifiers and conditions, handle skip strategies, time constraints, and select results, with complete Java and Scala code examples for complex event processing.
This article explains the evolution of Spark SQL from Shark, describes its key features such as SchemaRDD and in‑memory columnar storage, compares row‑based and column‑based storage, and provides practical Scala code examples for creating DataFrames and loading data from various sources.
This article presents a complete solution for real‑time advertising click counting using Spark Structured Streaming combined with Redis Streams, detailing the business scenario, data flow, input/output formats, and step‑by‑step implementation including data extraction, processing, storage, and query via Spark‑SQL.
To improve the performance of writing streaming data to Kafka, the article demonstrates how to replace per-partition KafkaProducer creation with a lazily-initialized, broadcasted producer in Scala, reducing overhead and achieving dozens‑fold speed gains.
This article provides a detailed introduction to Hadoop's ecosystem—including its core modules (Common, HDFS, YARN, MapReduce), the design of a high‑availability HDFS cluster, the principles of distributed file systems, and a complete Scala WordCount MapReduce program—offering a solid foundation for big‑data practitioners.
Aloha is a Scala‑implemented distributed scheduling framework built on Spark that provides extensible plugins, high‑availability master/worker architecture, REST submission, custom application interfaces, event listeners, and a Scala‑based RPC system for managing long‑running tasks such as Spark, Flink, and ETL jobs.
This article provides a comprehensive tutorial on Apache Flink's Table API, explaining its concepts, core features, and a wide range of operators such as SELECT, WHERE, GROUP BY, UNION, JOIN, and various window functions, while offering complete Scala code examples, custom sources, sinks, and an end‑to‑end job that computes page‑view counts per region using event‑time tumbling windows.
This article explains Scala extractors, showing how the apply and unapply methods work, provides code examples for email address extraction, and demonstrates their use in pattern‑matching scenarios.
This resource provides a curated collection of links to a series of Scala tutorial articles covering installation, basic syntax, data types, variables, access modifiers, operators, control structures, functions, collections, traits, pattern matching, and more, offering a comprehensive learning path for developers.
This article explains Scala's powerful pattern matching mechanism, demonstrates matching on integers, strings, and types with code examples, compares it to Java's switch, and introduces case classes as special classes optimized for pattern matching, highlighting their automatically generated methods and usage.
This article explains Scala traits, how they extend Java interfaces with concrete method implementations, demonstrates their syntax and usage through code examples, and details the trait construction order during object initialization.
