Configuring Kerberos‑Enabled HDFS Access with Maven in a Hadoop Cluster

Before following the steps, assume the reader is familiar with Java, Maven, HDFS, and Kerberos.

Implementation Overview

The final directory structure is shown in the included diagram, and a new Maven project should be created with the following

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.6.5</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.6.5</version>
</dependency>

added to pom.xml.

Copy the krb5.conf file from the KDC to the client machine and generate a keytab with the command:

kadmin.local:  xst -norandkey -k deng_yb.keytab [email protected]

Download the core-site.xml configuration file and place it on the client.

HDFSUtils Java Class

The utility class loads the HDFS configuration, sets the Kerberos system properties, logs in using the keytab, and provides a helper method to resolve resource paths:

package deng.yb.hdfsUtils;

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;

public class HDFSUtils {
    /** Load HDFS client configuration */
    public static Configuration initConfiguration() {
        Configuration configuration = new Configuration();
        configuration.addResource(new Path(getPath("core-site.xml")));
        return configuration;
    }

    /** Initialize Kerberos environment */
    public static void initKerberosENV(Configuration conf) {
        System.setProperty("java.security.krb5.conf", getPath("krb5.conf"));
        System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
        try {
            UserGroupInformation.setConfiguration(conf);
            UserGroupInformation.loginUserFromKeytab("[email protected]", getPath("deng_yb.keytab"));
            System.out.println(UserGroupInformation.getCurrentUser());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static String getPath(String file) {
        if (file == null) return null;
        return Thread.currentThread().getContextClassLoader().getResource(file).getPath();
    }
}

Test Application

The App class demonstrates how to initialize the configuration, set up Kerberos, and list files in the HA nameservice:

package deng.yb.hdfsUtils;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class App {
    public static void main(String[] args) {
        Configuration configuration = HDFSUtils.initConfiguration();
        HDFSUtils.initKerberosENV(configuration);
        try {
            Path dstPath = new Path("hdfs://nameservice1/");
            FileSystem fileSystem = FileSystem.get(configuration);
            FileStatus[] listStatus = fileSystem.listStatus(dstPath);
            for (FileStatus fileStatus : listStatus) {
                Path path = fileStatus.getPath();
                System.out.println(path);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Running the main method prints the file paths in the HDFS root, as shown in the screenshot.

Call to Action

Feel free to like, bookmark, and share the article.