100 Linux Interview Questions for Big Data Operations

Linux Basic Commands (20 Questions)

The first section covers fundamental Linux commands essential for daily big data operations. Each question includes the command syntax, common options, and practical usage scenarios in Hadoop, Spark, and Kafka environments.

File System Navigation

ls, ls -l, ls -a : List directory contents; ls -l shows detailed permissions, ownership, size, and modification time (used to check config file permissions like hdfs-site.xml); ls -a reveals hidden files such as .bashrc for environment variables.

cd, cd .., cd ~ : Change directory; cd .. moves to parent directory; cd ~ returns to user home (e.g., /home/hadoop). Frequently used to switch between /opt/hadoop/etc/hadoop (config) and /var/log (logs).

pwd : Print working directory absolute path. Critical in complex cluster directory structures (e.g., /data1/hdfs/data, /data2/kafka/logs) to avoid path errors.

mkdir, mkdir -p : Create directories; -p recursively creates parent directories. Used for HDFS directory creation like /data/hdfs/name and /data/hdfs/data.

rm, rm -f, rm -r : Remove files/directories; -f forces deletion without prompt; -r recursively deletes directories. Warning: deleting Hadoop data directories (e.g., /data/hdfs/data) requires cluster shutdown and backup to prevent data loss.

cp, cp -r : Copy files/directories; -r recursively copies entire directory trees. Common for backing up configs: cp -r /opt/hadoop/etc/hadoop/* /backup/.

mv : Move or rename files/directories. Unlike cp, mv does not preserve the source (cut-paste). Used for log rotation ( mv namenode.log namenode.log.202409) and moving configs for version comparison.

touch : Create empty files or update timestamps. Used for permission testing ( touch test.txt) or marking process state via PID file timestamps.

File Viewing and Searching

cat, more, less : View file content. cat dumps entire file; more pages forward only; less supports bidirectional scrolling and search. Recommendation : less for large logs (e.g., 10GB namenode.log) with search ( /ERROR).

head -n 10, tail -n 10, tail -f : Show first/last 10 lines; tail -f monitors real-time appends. Used for config headers, recent logs, and live task log tracking ( tail -f yarn.log).

echo, echo $PATH : Output strings or variable values. echo $JAVA_HOME verifies JDK environment for Hadoop/Spark command discovery.

which java, whereis java : which locates executable in PATH; whereis finds all related files (binary, source, man pages). Helps diagnose JDK version conflicts.

find / -name "hadoop*.xml" : Recursively search for files matching pattern. Locates misplaced Hadoop config files.

grep, grep -i, grep -r : Search text patterns. -i ignores case; -r recurses directories. Example: grep -r "Exception" /opt/spark/logs to locate Spark job failures.

wc, wc -l, wc -w, wc -c : Count lines, words, bytes. wc -l /var/log/hadoop/hdfs/*.log estimates log growth rate.

Archiving, Permissions, and Disk Usage

tar -zcvf, tar -zxvf : Compress ( -zcvf) and extract ( -zxvf) gzip tarballs. Deploy components ( tar -zxvf spark-3.3.0.tgz) and backup configs ( tar -zcvf hadoop-conf.tar.gz /opt/hadoop/etc/hadoop).

chmod 755, chmod u+x : Change permissions. 755 = owner rwx, group/others r-x. Scripts like start-all.sh need execute permission; configs typically 644.

chown -R hadoop:hadoop /opt/hadoop : Recursively change owner/group. Hadoop components must run as hadoop user, not root, to avoid permission issues (e.g., unable to write logs).

df -h, du -sh : df -h shows filesystem capacity/usage; du -sh /var/log shows directory size. Used to detect disk pressure and identify large directories.

free -h : Display memory usage. Monitor for memory leaks or OOM during Spark job execution.

Permissions & User Management (10 Questions)

Covers permission bits, directory execute meaning, file types, symbolic vs numeric modes, group management, user creation, privilege escalation, sudoers, sticky bit, and umask.

r/w/x on directories : x allows cd into directory. HDFS data directories need x for DataNode block access.

ls -l first char : d = directory, - = regular file, l = symlink, b = block device, c = char device. Distinguish /dev/sda (disk) from /etc/hosts (file).

chmod u+x vs chmod 700 : u+x adds execute for owner only; 700 sets owner rwx and removes all other permissions. Fine-grained control is safer.

chgrp : Change group ownership. Unify component directories under hadoop group for shared team access.

useradd, passwd : Create dedicated service users ( hadoop, kafka) to avoid root security risks.

su - hadoop vs sudo -u hadoop command : su switches user session; sudo -u runs single command as target user. Prefer sudo for efficiency (e.g., sudo -u hdfs hdfs dfs -ls /).

visudo : Safely edit sudoers. Grant passwordless execution for specific cluster commands (e.g., hadoop ALL=(ALL) NOPASSWD: /opt/hadoop/bin/*).

id, id -g, id -G : Show UID, GID, and group memberships. Verify user belongs to hadoop group.

sticky bit (chmod +t /tmp) : Only file owner or root can delete files in directory. Protects shared temp directories like /tmp/hadoop.

umask 022 : Default file permission 644 ( 666-022), directory 755 ( 777-022). Ensures secure defaults for new configs and directories.

Process & Service Management (15 Questions)

Focuses on process inspection, resource monitoring, termination, service control, port inspection, and Java-specific tools.

ps -ef | grep java : List all Java processes. Identify Hadoop/Spark daemons (NameNode, ResourceManager).

top : Real-time CPU/memory monitoring. Press P (CPU sort), M (memory sort), k (kill by PID). Spot resource-hungry Spark tasks.

kill -9 PID : Force terminate. Use cautiously on stuck MapReduce tasks; may cause data inconsistency.

pkill java, killall java : Bulk kill all Java processes. Clean up residual processes before cluster restart.

systemctl start/stop/restart/status : Manage systemd services (e.g., hadoop-hdfs-namenode, kafka).

systemctl enable/disable : Toggle auto-start on boot. Essential for core services (ZooKeeper, HDFS) to recover automatically after node reboot.

service vs systemctl : systemctl is modern systemd replacement with richer features (dependencies, logs). Recommended for production.

netstat -tuln | grep 8088 : Check listening TCP/UDP ports. Diagnose Address already in use errors for YARN WebUI (8088).

lsof -i :9000 : Show process holding port 9000 (HDFS NameNode RPC).

jps : List Java PIDs and main class names. Quick cluster health check (NameNode, DataNode, ResourceManager) — cleaner than ps.

nohup command & : Run immune to hangups, output to nohup.out. Standard for launching Kafka, Flink daemons.

jobs, fg %n : List background jobs; bring to foreground. Monitor start-dfs.sh background execution.

bg : Resume suspended job in background. Recover from accidental Ctrl+Z.

pstree : Tree view of parent-child processes. Trace YARN child processes (MapReduce tasks).

pgrep -f "namenode" : Output PIDs matching pattern. Script-friendly for bulk actions: kill $(pgrep -f "namenode").

Network Configuration (10 Questions)

Covers interface inspection, connectivity testing, routing, hostname management, host resolution, secure copy, SSH, HTTP checks, downloads, and connection tracking.

ifconfig / ip addr : View/configure network interfaces. Verify node IPs (e.g., 192.168.1.101) to prevent communication failures.

ping -c 4 bigdata02 : Test connectivity with 4 ICMP packets. Confirm NameNode-DataNode reachability.

traceroute bigdata03 : Show network path. Locate faulty hops (e.g., switch failure).

hostname, hostnamectl set-hostname bigdata01 : View/set persistent hostname. Standardize node names ( bigdata01, bigdata02) for cluster identification.

/etc/hosts : Local hostname-to-IP mapping. Mandatory for component communication via hostnames (e.g., hdfs://bigdata01:9000) without external DNS dependency.

scp core-site.xml hadoop@bigdata02:/opt/hadoop/etc/hadoop/ : Secure copy config files across nodes. Synchronize cluster configs; large-scale sync uses rsync or Ansible.

ssh bigdata02, ssh -p 2222 bigdata02 : Remote login. Specify non-standard SSH port with -p.

curl http://bigdata01:50070 : HTTP probe. Verify HDFS NameNode WebUI (50070), YARN (8088), Spark (4040) availability.

wget URL : Download files (supports resume -c). Fetch Hadoop/Spark tarballs from Apache mirrors.

netstat -an | grep ESTABLISHED : List established TCP connections. Confirm Kafka producer/consumer connections to broker (9092).

Log & File Processing (15 Questions)

Demonstrates log filtering, compression handling, stream editing, column extraction, sorting, deduplication, cleanup, remote backup, splitting, symlinks, disk testing, file typing, line-ending conversion, and output tee.

grep "ERROR" log.txt | wc -l : Count error lines. Gauge failure severity.

grep -v "INFO" log.txt : Exclude INFO lines. Focus on WARN/ERROR: grep -v "INFO" yarn.log | grep -E "WARN|ERROR".

zgrep "ERROR" app.log.gz : Search compressed logs without decompression. Analyze historical logs ( namenode.log.202409.gz).

sed 's/old/new/g' file.txt : Global replace. Modify configs in-place:

sed -i 's/dfs.replication=3/dfs.replication=2/g' hdfs-site.xml

.

awk '{print $1, $3}' data.txt : Print columns 1 and 3. Extract timestamp and URL from access logs.

sort vs sort -n : Lexicographic vs numeric sort. Numeric sort required for execution times.

uniq -c : Count adjacent duplicates. Pipeline for top errors: grep "ERROR" log.txt | sort | uniq -c | sort -nr | head -10.

find /var/log -name "*.log" -mtime +7 -delete : Delete logs older than 7 days. Schedule via crontab to prevent disk exhaustion.

tar -zcvf - /var/log | ssh bigdata02 "tar -zxvf - -C /backup" : Stream compressed logs to remote backup.

split -l 1000 largefile.txt part_ : Split large files into 1000-line chunks. Enable parallel processing of 10GB logs.

ln -s /opt/hadoop current-hadoop : Symlink for version switching. Point current-hadoop to active version ( hadoop-3.3.4 or hadoop-3.4.0).

dd if=/dev/zero of=/data/test bs=1G count=1 : Create 1GB test file. Benchmark disk I/O for HDFS data directories.

file hadoop : Identify file type (text, ELF binary, archive). Distinguish scripts from binaries.

dos2unix script.sh : Convert Windows CRLF to Unix LF. Fix "command not found" errors from Windows-edited scripts.

command | tee output.txt : Duplicate output to screen and file. Capture hdfs dfsadmin -report for later analysis.

Shell Scripting & Automation (10 Questions)

Covers shebang, execution permissions, variables, positional parameters, conditionals, loops, cron, date formatting, and comments.

#!/bin/bash : Declare bash interpreter. Avoids syntax incompatibilities (e.g., arrays).

chmod +x script.sh; ./script.sh : Add execute bit and run. Required for automation scripts (bulk service start, health checks).

name="hadoop"; echo $name : Variable assignment (no spaces around =) and reference. Store paths like HADOOP_HOME="/opt/hadoop" for maintainability.

$0, $1, $#, $? : Script name, first argument, argument count, last exit code. Build parameterized deploy scripts ( ./deploy.sh node1 3).

if [ -d "/opt/hadoop" ]; then ... fi : Directory existence check. Prevent invalid command execution.

for node in ${nodes[@]}; do ... done : Iterate array of hostnames. Batch operations: SSH port checks, file distribution.

while read line; do ... done < file.txt : Line-by-line file processing. Read node lists: while read node; do ssh $node "command"; done < nodes.txt.

crontab -e; 0 3 * * * /opt/scripts/clean_logs.sh : Schedule daily 3 AM log cleanup. Automate repetitive maintenance.

$(date +%Y%m%d) : Current date as

20240925</strong>. Archive logs with date stamps: <code>namenode.log.$(date +%Y%m%d)

.

# comment : Single-line comments. Document purpose, parameters, warnings (e.g., # Backup Hadoop configs, run after cluster stop).

Big Data Operations Specific (20 Questions)

Addresses cluster-specific tasks: passwordless SSH, firewall, time sync, environment variables, system info, CPU/disk tuning, swap, kernel params, and component CLI usage.

Passwordless SSH : ssh-keygen -t rsa -P "" then ssh-copy-id hadoop@bigdata02. Required for start-dfs.sh to launch remote daemons without password prompts.

Firewall : systemctl stop firewalld; systemctl disable firewalld. Big data components use many ports (HDFS 50010, Kafka 9092); firewall blocks inter-node communication.

NTP time sync : yum install ntp, sync all nodes to single NTP server. Distributed systems rely on consistent timestamps for HDFS block metadata, Kafka message ordering, and task scheduling.

echo $HADOOP_HOME, which hadoop : Verify Hadoop installation path. HADOOP_HOME misconfiguration breaks hadoop command.

source /etc/profile : Reload environment variables after editing JAVA_HOME, HADOOP_HOME without terminal restart.

cat /etc/redhat-release, lsb_release -a : Identify OS version. Determines package manager ( yum for CentOS, apt for Ubuntu).

lscpu : Show CPU cores/threads. Guide spark.executor.cores setting (typically half of physical cores).

mount /dev/sdb1 /data; /etc/fstab entry : Permanent mount for HDFS DataNode and Kafka log directories. Ensures persistence across reboots.

swapoff -a : Disable swap. Swap latency degrades Spark/Flink in-memory performance; production clusters disable it.

sysctl -w vm.max_map_count=262144 : Increase memory map areas. Prevents Elasticsearch "max virtual memory areas" errors.

hdfs getconf -confKey dfs.replication : Query effective replication factor (default 3). Balances storage cost vs availability.

hdfs dfs -ls / : List HDFS root. "Connection refused" indicates NameNode down, port 9000 conflict, or core-site.xml misconfig.

yarn application -list; yarn application -kill <AppID> : List/kill YARN apps. Terminate stuck or greedy MapReduce/Spark jobs.

kafka-topics.sh --list --bootstrap-server bigdata01:9092 : Enumerate Kafka topics. Verify broker health and topic creation.

zkCli.sh -server bigdata01:2181 : ZooKeeper CLI. Inspect stored metadata: HDFS HA state, YARN resource info, Kafka partition metadata.

hbase shell; list : HBase interactive shell. Manage tables, verify HBase/MetaStore status.

spark-submit --class MainClass --master yarn app.jar : Submit Spark jobs. Required params: main class, cluster master, JAR path. Tuning params directly affect performance.

flink run -p 10 job.jar : Submit Flink job with parallelism 10. Match parallelism to CPU cores.

hive; show databases; : Hive CLI. List databases, validate Metastore connectivity.

Ansible : Agentless automation via SSH. Batch command execution, config distribution, service restarts. Core advantage: no client installation on target nodes.

Summary

The 100 questions span the full Linux operational spectrum for big data platforms. Each answer pairs command syntax with real-world cluster scenarios — explaining not just what the command does, but why it matters in Hadoop, Spark, Kafka, HBase, Flink, and ZooKeeper contexts. The emphasis on practical reasoning (e.g., why disable swap, why use less over cat, why umask 022) prepares candidates to diagnose and resolve production issues rather than merely recite flags.