100 Advanced Linux Interview Questions for Big Data Operations

Advanced Commands and Text Processing (15 Questions)

The first section focuses on Linux command-line proficiency for big data log analysis and system administration. Key examples include:

Question 1: Using awk to count block reports per DataNode from HDFS logs (e.g.,

awk '/reported [0-9]+ blocks/ {print $4}' hdfs.log | sort | uniq -c

). The analysis explains extracting the 4th column (hostname) and counting occurrences to assess DataNode health.

Question 2: sed -i 's/^#//g' /etc/hosts removes all comment lines; an advanced variant sed -i '/^#bigdata03/s/^#//g' /etc/hosts enables only specific node entries during cluster expansion.

Question 3:

find /opt/hadoop/etc/hadoop -name "*.xml" -mtime 0 -exec cp {} /backup/ \;

backs up Hadoop XML configs modified within 24 hours for rollback readiness.

Question 4: grep -E "ERROR|WARN" app.log | awk '{print $1,$2,$NF}' extracts timestamps and error messages for faster debugging.

Question 5: df -h | awk '$5 > 80 {print $0}' identifies partitions exceeding 80% usage; combined with crontab for automated alerts.

Question 6: tar -zcvf hadoop.tar.gz --exclude=logs /opt/hadoop excludes bulky log directories during backup.

Question 7: lsof | grep deleted finds deleted files still held by processes (e.g., HDFS logs), causing unreleased disk space; lsof -p $(pgrep -f cn) inspects StarRocks CN process file handles.

Question 8: ss -tanp | grep 9092 | wc -l counts Kafka broker TCP connections; high counts indicate client connection leaks.

Question 9: ps -eo %cpu,%mem,cmd --sort=-%cpu | head -10 pinpoints top CPU/memory consumers like runaway Spark jobs.

Question 10:

rsync -avz --delete /opt/hadoop/ hadoop@bigdata02:/opt/hadoop/

ensures consistent cluster configs via archive, verbose, compressed sync with deletion of extraneous files.

Question 11: zcat access.log.*.gz | grep "POST /api" | wc -l aggregates API call counts across compressed logs.

Question 12: stat output distinguishes Access (last read), Modify (content change), Change (metadata change); Modify time guides cold-data migration decisions.

Question 13: diff -u old.conf new.conf > patch.diff and patch old.conf < patch.diff enable version-controlled config distribution.

Question 14: watch -n 5 "jps | grep -v Jps" refreshes Java process list every 5 seconds to monitor NameNode/StarRocks FE stability.

Question 15: find / -type f -perm 777 -exec ls -l {} \; detects world-writable files (e.g., core-site.xml) posing security risks.

System Performance Tuning (15 Questions)

This section dives into kernel parameters, resource limits, and monitoring tools critical for big data workloads:

Question 16: vm.swappiness controls swap aggressiveness (0–100). Recommended value 1 (not 0) for Hadoop/Spark/Kafka: avoids swap-induced latency spikes (10–100×) while retaining a safety buffer against OOM killer terminating critical processes during memory spikes. Setting sysctl -w vm.swappiness=1 persists via /etc/sysctl.conf.

Question 17: ulimit -n sets max open file descriptors. Kafka Broker and HDFS DataNode handle thousands of files (partition logs, data blocks); default 1024 causes "too many open files" errors. Raise to 65535+.

Question 18: Permanent limits require editing /etc/security/limits.conf ( * soft nofile 65535, * hard nofile 65535) and /etc/systemd/system.conf ( DefaultLimitNOFILE=65535) followed by reboot.

Question 19: vm.max_map_count limits memory map areas. Elasticsearch needs 262144 (vs. default 65530) for heavy mmap usage (index shards). Set via sysctl -w vm.max_map_count=262144.

Question 20: net.ipv4.tcp_tw_recycle (fast TIME_WAIT recycle) and net.ipv4.tcp_tw_reuse (reuse sockets). For high-concurrency Kafka producers, enable sysctl -w net.ipv4.tcp_tw_reuse=1 to prevent port exhaustion; tw_recycle is deprecated due to NAT issues.

Question 21: iostat -x 5 shows %util (device busy percentage). Sustained >90% signals disk bottleneck (e.g., HDFS DataNode write lag); remedy: add disks or migrate to SSD.

Question 22: sar -n DEV 5 reports rxkB/s and txkB/s (KB/s received/sent). Approaching NIC capacity (e.g., 125 MB/s for 1 Gbps) indicates network saturation.

Question 23: mpstat -P ALL 5 reveals per-core CPU usage. A single core at 100% %usr during Spark runs suggests single-threaded bottlenecks (e.g., inefficient serialization).

Question 24: buff/cache in free -h is reclaimable kernel file cache. Release via echo 3 > /proc/sys/vm/drop_caches but avoid in production—cache boosts performance; use only for temporary memory diagnostics.

Question 25: I/O schedulers: HDD → deadline (latency-optimized), SSD/NVMe → noop (minimal CPU overhead) or mq-deadline / kyber for multi-queue devices. Check current with cat /sys/block/sda/queue/scheduler; set via echo mq-deadline > /sys/block/sda/queue/scheduler.

Question 26: nice (launch priority) and renice (adjust running). Example: renice -10 $(pgrep -f NodeManager) elevates YARN NodeManager priority to prevent resource starvation.

Question 27: sysctl -w net.core.somaxconn=1024 expands TCP listen queue. Kafka Broker defaults (128) drop connections under load; 1024+ improves concurrent acceptance.

Question 28: Load average (1/5/15 min) from uptime / top reflects runnable + uninterruptible (I/O wait) processes. Compare to CPU cores (N): load ≈ N = saturated; load > N = queuing (e.g., 4-core with load 10 = 6 processes waiting).

Question 29: tc qdisc add dev eth0 root netem delay 100ms injects 100 ms latency to test HDFS replica sync resilience under network degradation.

Question 30: xfs preferred over ext4 for DataNode/Kafka directories: supports larger files, higher IOPS, superior journaling for large-file, high-concurrency write workloads.

Cluster Advanced Configuration and Management (20 Questions)

Covers Hadoop, YARN, Kafka, Spark, Flink, HBase, Zookeeper, Elasticsearch, Hive tuning:

HDFS Federation (31): Multiple NameNodes (independent namespaces) sharing DataNodes via dfs.nameservices to overcome single NameNode memory limits.

Trash Interval (32): fs.trash.interval=1440 (minutes) in core-site.xml enables 24-hour recovery window for accidental deletes.

YARN Capacity Scheduler (33): Queue root.prod gets 60% guaranteed ( capacity=60) up to 80% ( maximum-capacity=80) of cluster memory.

Spark Dynamic Allocation (34): Enable via spark.dynamicAllocation.enabled=true, spark.shuffle.service.enabled=true, minExecutors=2, maxExecutors=20 for workload-adaptive executor scaling.

Kafka Log Retention (35): log.retention.hours (default 168) sets message TTL; log.segment.bytes (default 1 GB) controls segment size. Balance: shorten retention for hot topics, increase segment size for cold topics to reduce file count.

HBase Region Split (36): hbase.hregion.max.filesize (default 10 GB) triggers region split. Too small → frequent splits → ZooKeeper metadata bloat → performance degradation.

Flink Checkpoint vs Savepoint (37): Checkpoint (auto, env.enableCheckpointing(60000)) for failure recovery; Savepoint (manual, bin/flink savepoint <jobId> <path>) for version upgrades.

ZooKeeper Timeouts (38): tickTime (base ms, default 2000), initLimit (follower sync timeout = tickTime×initLimit), syncLimit (heartbeat timeout = tickTime×syncLimit).

Elasticsearch Sharding (39): Target 10–20M docs per shard. For 10M docs, 5 primary shards ( index.number_of_shards=5) with 1 replica ( index.number_of_replicas=1) balances query parallelism and overhead.

Hive Dynamic Partition (40): hive.exec.dynamic.partition=true + mode=nonstrict allows fully dynamic partition creation for bulk multi-partition inserts.

Hadoop Rack Awareness (41): Script defined by net.topology.script.file.name returns rack paths (e.g., /rack1). Ensures cross-rack replica placement for durability and rack-local YARN scheduling to reduce network traffic.

Kafka Cleanup Policy (42): delete (time/size-based, for logs) vs compact (retain latest key, for KV stores like user configs).

Spark Memory Split (43): spark.driver.memory (scheduler, 4–16 GB) vs spark.executor.memory (compute, ~70% node memory, e.g., 80 GB on 128 GB node).

HBase RegionServer Memory (44): HBASE_HEAPSIZE in hbase-env.sh (e.g., 32 GB). hbase.regionserver.global.memstore.upperLimit=0.4 caps MemStore at 40% heap; exceeding forces flush to avoid OOM.

YARN Allocation Limits (45): yarn.nodemanager.resource.memory-mb = total allocatable (e.g., 128 GB); yarn.scheduler.maximum-allocation-mb ≤ that, caps single container request.

Flink RocksDB State Backend (46): state.backend: rocksdb + state.checkpoints.dir: hdfs:///flink-checkpoints enables large-state streaming jobs beyond memory limits.

Hive Engine Choice (47): Tez excels at complex DAGs (multi-join); Spark suits iterative ML. Switch via set hive.execution.engine=spark; or hive-site.xml.

Kafka Partition/Replication (48): Defaults: num.partitions=1, default.replication.factor=1. Production: partitions = brokers × 3 for load balance; replication = 3 for HA.

ZooKeeper Quorum (49): server.1=ip1:2888:3888 in zoo.cfg. Majority (⌈N/2⌉) must survive. Odd nodes (3,5) optimal: 3 tolerates 1 failure, 4 also tolerates 1 (wasted node).

DataNode Disk Tolerance (50): dfs.datanode.failed.volumes.tolerated=1 (for 4-disk node) allows single disk failure without DataNode shutdown.

Complex Troubleshooting (20 Questions)

Real-world failure diagnosis with concrete commands and root-cause analysis:

HDFS Corrupt Blocks (51): hdfs fsck / | grep CORRUPT locates damaged files; hdfs fsck /path -delete removes corrupt blocks (healthy replicas auto-replicate); restore from backup if no replicas.

YARN Container Exit 143 (52): SIGTERM from ResourceManager due to memory overrun. Fix: increase spark.executor.memory or mapreduce.map.memory.mb, or optimize job memory footprint.

Kafka OffsetOutOfRange (53): Consumer offset deleted (exceeded log.retention). Reset via --from-beginning or kafka-consumer-groups.sh --reset-offsets to earliest/latest.

Spark Data Skew (54): top shows one Executor at 90% CPU; jstack <pid> reveals hot-key processing threads.

HBase Scan Timeout (55): Causes: RegionServer overload, network latency, huge tables. Diagnose: top (RS resources), ping (network), hbase shell count (table size).

Flink Checkpoint Timeout (56): Checkpoint expired before completing (default 10 min). Increase state.checkpoint.timeout, enable RocksDB compression, add TaskManager resources.

ZooKeeper Split-Brain (57): Network partition triggers multiple leaders. Mitigate: quorumListenOnAllIPs=true, stable networking, odd node count.

Hive Silent Failure (58): yarn logs -applicationId <appId> for container logs; systemctl status hadoop-yarn-nodemanager; hdfs dfs -df -h / for space.

Kafka Disk IO 100% (59): Temporary: reduce retention on low-priority topics (

kafka-topics.sh --alter --topic slow_topic --config retention.ms=86400000

). Permanent: add disks to log.dirs or migrate topics to new brokers.

DataNode BindException (60): netstat -tuln | grep 50010 or lsof -i :50010 finds port hog; kill or change dfs.datanode.address.

Spark Streaming Kafka Lag (61):

kafka-consumer-groups.sh --describe --group spark-group --bootstrap-server localhost:9092

; lag = SUM(LOG-END-OFFSET - CURRENT-OFFSET) per partition.

HBase OOM (62): Causes: MemStore pressure, oversized BlockCache, too many regions. Tune: lower hbase.regionserver.global.memstore.upperLimit, reduce hfile.block.cache.size, merge small regions.

NodeManager Heartbeat Timeout (63): ping/traceroute <RM-IP> for connectivity; free -h / iostat -x for resource saturation; increase yarn.nodemanager.heartbeat.interval-ms (default 3s).

Kafka LeaderNotAvailable (64): Partition leader broker down, no ISR replica. Restart broker or run kafka-preferred-replica-election.sh.

NameNode Stuck Loading fsimage (65): Corrupt/oversized fsimage (metadata bloat). Copy healthy fsimage from Standby or enable Federation.

Flink Backpressure (66): WebUI shows Backpressure column; jstack <TaskManager-PID> reveals mass WAITING threads.

Hive Metastore MySQL Connections (67): Raise max_connections=1000 in my.cnf; set javax.jdo.option.ConnectionPoolMaxSize in hive-site.xml.

Spark Low CPU Utilization (68): Causes: poor data locality (increase spark.locality.wait), shuffle skew, insufficient executors.

ZooKeeper Connection Refused (69): Verify server.x IPs/ports in zoo.cfg, matching myid files, open 2888/3888 through firewall.

Elasticsearch CircuitBreaker (70): Request memory > 70% heap limit. Raise indices.breaker.total.limit=80%; optimize queries (field selection, pagination).

Automation and Monitoring (15 Questions)

Scripting, configuration management, and observability stacks:

Ansible JDK Deployment (71): Playbook unarchives JDK to /opt, appends JAVA_HOME / PATH to /etc/profile via lineinfile with with_items.

Prometheus+Grafana HDFS Monitoring (72): Deploy prometheus-hadoop-exporter, configure scrape_configs, import Grafana HDFS dashboard.

ZooKeeper Auto-Restart Script (73): Bash loop over nodes: ssh $node "pgrep -f QuorumPeerMain | wc -l", restart via systemctl start zookeeper if count=0.

ELK for Kafka Logs (74): Filebeat → Logstash (filter/extract level, timestamp) → Elasticsearch → Kibana.

Cron HDFS Backup (75):

0 2 * * * /opt/scripts/backup_hdfs.sh >> /var/log/hdfs_backup.log 2>&1

(2 AM daily, merged stderr/stdout).

Python HDFS Corrupt Block Alert (76): subprocess.run(["hdfs","dfsadmin","-report"]), parse "Corrupt blocks" line, SMTP email if count>0.

Grafana YARN Alert to Slack (77): Panel metric yarn_cluster_resource_used_percent, threshold >90% for 5m, Slack webhook notification.

Ansible Kafka Config Templating (78): server.properties.j2 with

broker.id={{ inventory_hostname | regex_replace('bigdata','') }}

; Playbook template module generates per-node config.

Systemd Spark Job Service (79): Unit file with User=spark, ExecStart=spark-submit ..., Restart=always, dependencies on hadoop.service / kafka.service.

PromQL HDFS Write Rate (80): rate(dfs_namenode_write_bytes[1h]) computes 1-hour average write throughput.

HDFS Directory Size Top-10 (81):

hdfs dfs -du -s /* | sort -k1,1nr | head -10 | awk '{printf "%.2f GB\t%s
", $1/1024/1024/1024, $2}'

.

Logstash Drop INFO Logs (82): Filter: if [level] == "INFO" { drop {} } retains only WARN/ERROR.

Ansible DataNode Rolling Restart (83): Playbook stops/starts hadoop-hdfs-datanode service, verifies via jps | grep DataNode with failed_when.

Grafana Spark Shuffle Trends (84): Enable spark.metrics.conf (Graphite/Prometheus); plot spark_job_shuffle_read_bytes and spark_job_shuffle_write_bytes as time-series lines grouped by job ID.

Kafka Lag Alert to WeCom (85): Bash computes lag via kafka-consumer-groups.sh; if >1M, POST JSON to WeCom webhook.

Security and Compliance (10 Questions)

Access control, auditing, encryption, and hardening:

HDFS Permissions (86): dfs.permissions.enabled=true (default). Grant access: hdfs dfs -chown user:group /path, hdfs dfs -chmod 750 /path.

Auditd Config Monitoring (87): auditctl -w /opt/hadoop/etc/hadoop -p rwxa -k hadoop-config watches read/write/execute/attribute changes.

Hadoop Kerberos Setup (88): 1) Deploy KDC, create principals ( hdfs/[email protected]). 2) Generate/distribute keytabs. 3) Enable in core-site.xml with keytab paths.

Immutable Config Files (89): chattr +i /opt/hadoop/etc/hadoop/core-site.xml prevents modification even by root.

Firewall Port Whitelisting (90): firewall-cmd --add-port=50070/tcp --permanent (NameNode UI), 8088/tcp (YARN UI), 9000/tcp (HDFS RPC), then --reload.

Apache Ranger Column-Level Hive (91): Centralized policy engine; create Hive policy in Ranger UI specifying database, table, column, grant "select" to user/group.

Keytab Rotation Automation (92): KDC: ktutil create new keytab; scp to nodes; chmod 400, chown hdfs:hadoop; restart components.

SELinux Permissive Mode (93): Modes: Enforcing (block), Permissive (log only), Disabled. Big data clusters use Permissive to avoid false positives on cross-directory accesses while retaining audit logs.

HDFS TLS/SSL (94): Generate/distribute certs; set dfs.https.enable=true, keystore/truststore paths in core-site.xml / hdfs-site.xml; verify https://namenode:50470.

Spark Submission Audit (95): Enable yarn.audit.logger=INFO,RFA in yarn-site.xml; logs at /var/log/hadoop/yarn/yarn-audit.log; filter with grep "Submit Application".

Containerization and Cloud Environments (5 Questions)

Modern deployment patterns:

Docker HDFS DataNode (96): Match container UID to host data directory UID; limit resources ( --cpus 4 --memory 16g); bind-mount host disks ( -v /data/hdfs:/data/hdfs).

Kubernetes Spark Executor Resources (97): SparkApplication spec: executor requests: {cpu: "1", memory: "4g"}, limits: {cpu: "2", memory: "8g"}.

AWS EMR vs On-Prem (98): Elastic scaling vs manual provisioning; S3 vs HDFS (consistency/performance trade-offs); managed control plane vs self-managed ops.

Docker Compose Zookeeper/Kafka/Flink (99): docker-compose.yml with services: zookeeper (cp-zookeeper, port 2181), kafka (cp-kafka, depends_on zookeeper, KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181), flink-jobmanager / flink-taskmanager (flink image, port 8081, depends chain).

Kubernetes HBase Monitoring (100): Deploy Prometheus Operator + ServiceMonitor for HBase Exporter; PodMonitor for container metrics (CPU/memory/network); Grafana with mixed HBase/K8s dashboards.