130 Essential Big Data and Distributed Systems Interview Questions

This article compiles 130 interview questions covering big data technologies, distributed systems, and fundamental computer science topics, intended to assist candidates in preparing for technical interviews.

1. Difference between HashMap and Hashtable

2. Java garbage collection mechanisms and lifecycle

3. How to solve Kafka data loss issues

4. How Zookeeper ensures data consistency

5. Methods to handle memory overflow in Hadoop and Spark processing

6. Implement quicksort in Java

7. Design a database table structure for WeChat group red‑packet distribution (including table name, field names, and types)

8. Selection criteria: business scenario, performance requirements, maintenance and scalability, cost, open‑source activity

9. Spark tuning techniques

10. Differences and similarities between Flink and Spark communication frameworks

11. Java proxy mechanisms

12. Java memory overflow and memory leak concepts

13. Hadoop components and YARN schedulers

14. Hadoop shuffle process

15. Brief description of Spark cluster deployment modes

16. Performance comparison of reduceByKey vs. groupByKey in RDD

17. Brief description of HBase read/write process

18. Find the unique integer among 250 million integers when memory is insufficient to hold all numbers

19. Differences between CDH and HDP

20. Java atomic operations

21. Java encapsulation, inheritance, and polymorphism

22. JVM architecture model

23. Solution for Flume taildirSource duplicate data reads

24. How Flume guarantees no data loss

25. Java class loading process

26. Spark task execution principle

27. Write a thread‑safe singleton implementation

28. Design patterns overview

29. Use cases for Impala and Kudu, and their read/write performance characteristics

30. Kafka ACK mechanism principle

31. Ways to create indexes in Phoenix and their differences

32. Communication between Flink TaskManager and JobManager

33. Flink dual‑stream join methods

34. Flink state management and checkpoint workflow

35. Flink layered architecture

36. Flink windowing concepts

37. How Flink watermarks handle out‑of‑order data

38. Flink time handling

39. Flink support for exactly‑once sinks and sources

40. Flink job submission process

41. Differences between Flink connect and join

42. Strategies for restarting tasks

43. Hive locking mechanisms

44. Hive SQL optimization techniques

45. Hadoop shuffle process and architecture

46. How to optimize the shuffle process

47. Comparison of bubble sort and quicksort

48. Explanation of Spark stages

49. Differences between Spark mapPartitions and parallelize functions

50. Spark checkpointing process

51. Secondary sorting techniques

52. How to register a Hive UDF

53. SQL deduplication methods

54. Hive analytical and window functions

55. Hadoop fault tolerance when a node fails and recovers

56. Understanding JVM fundamentals

57. Java concurrency principles

58. Methods to implement multithreading

59. RocksDB state backend implementation (source‑level)

60. Differences among HashMap, ConcurrentMap, and Hashtable

61. How Flink checkpoints work and whether they affect operators or chains

62. Monitoring checkpoint failures

63. Differences between String, StringBuffer, and StringBuilder

64. Kafka storage process and reasons for high throughput

65. Examples of Spark optimization methods

66. Maximum parallelism of keyBy

67. Flink optimization techniques

68. Kafka ISR (in‑sync replica) mechanism

69. Four states of a Kafka partition

70. Seven states of a Kafka replica

71. Number of Flink TaskManagers

72. Performance comparison of if vs. switch and supported switch parameters

73. Kafka zero‑copy mechanism

74. Hadoop node fault‑tolerance mechanisms

75. HDFS replica placement strategy

76. Hadoop interview question collection (link provided)

77. Permission control for Kudu and Impala

78. Explanation of the TIME_WAIT state after a server closes a socket

79. What is exchanged during the three‑way handshake (SYN, ACK, SEQ, window size) and the four‑way termination

80. Differences between HashMap in JDK 1.7 and 1.8

81. Differences between ConcurrentHashMap in JDK 1.7 and 1.8

82. Kafka ACK details

83. SQL deduplication methods (GROUP BY, DISTINCT, window functions)

84. Hive SQL statements that cannot run on Spark SQL (link provided)

85. Scenarios that cause deadlocks

86. Transaction isolation levels (repeatable read, non‑repeatable read, read uncommitted, serializable)

87. Differences and similarities between Spark shuffle and Hadoop shuffle

88. Spark static memory vs. dynamic memory

89. Differences between MySQL B‑tree and hash indexes (B‑tree requires unique primary key; hash suitable for equality, not range queries)

90. Differences among UDF, UDTF, and UDAF

91. Hive SQL execution process

92. Spark SQL execution process

93. Find the top‑10 longest strings in an array

94. Flink data processing workflow

95. Comparison of Flink and Spark Streaming

96. Usage of Flink watermarks

97. Combining windows with streams

98. Real‑time alert design with Flink

99. Java topics: object‑orientation, containers, multithreading, singleton pattern

100. Flink topics: deployment, API, state, checkpoint, savepoint, watermark, restart strategies, DataStream operators, optimization, job and task states

101. Spark topics: principles, deployment, optimization

102. Kafka topics: read/write principles, usage, optimization

103. Hive external tables

104. Functional programming in Spark

105. Linear data structures and data structures overview

106. Spark mapping and RDD concepts

107. Java memory overflow and memory leak details

108. Multithreading implementation methods

109. Differences among HashMap, ConcurrentMap, and Hashtable

110. How Flink checkpoints work and whether they affect operators or chains

111. Monitoring checkpoint failures

112. Differences between String, StringBuffer, and StringBuilder

113. Kafka storage process and reasons for high throughput

114. Example Spark optimization methods

115. Maximum parallelism of keyBy

116. Flink optimization methods

117. Kafka ISR mechanism

118. States of a Kafka partition

119. States of a Kafka replica

120. Number of TaskManagers

121. Performance differences between if and switch statements

122. HDFS read/write process (explained with CAP theorem)

123. Principles for technology selection

124. Introduction to Kafka components

125. Differences between G1 and CMS garbage collectors

126. Discussion of the most familiar data structures

127. Methods to handle Spark OOM (out‑of‑memory) issues

128. Source code that has been studied

129. Spark task execution principles

130. The most challenging problem solved

131. HBase read/write process