Hadoop Word Count on AWS EC2: Step-by-Step Guide

Setting up distributed big data processing engines locally often presents memory limitations on developer workstations. Executing the Hadoop Word Count MapReduce application by deploying Apache Hadoop in Pseudo-Distributed Mode on an AWS EC2 instance bridges this gap, allowing software engineers to simulate multi-node cluster behaviors such as HDFS block replication, NameNode namespace management, and YARN resource orchestration all within a single cloud VM context.

Hadoop Cluster Setup Type

• Hadoop can be set up in different modes depending on how many machines are used and for what purpose. The most common Hadoop cluster setups are:
  • Standalone Mode
  • Pseudo-Distributed Mode
  • Fully Distributed Mode

Standalone Mode [Single Node]

• Standalone mode runs Hadoop without any cluster services. It is uses the local file system instead of HDFS.
  • No Name Node or Data Node daemons
  • No HDFS
  • Used mainly for learning and testing MapReduce logic

Pseudo-Distributed Mode (Pseudo Cluster)

• In pseudo-distributed mode, all Hadoop daemons run on a single machine, but they behave as if they are running in a real cluster. This is why it is called a pseudo cluster.
• Services Name that is running:
  • Name Node
  • Data Node
  • Resource Manager
  • Node Manager
  • Secondary Name Node
• It is uses HDFS , MapReduce and YARN. It is simulates a real Hadoop cluster and best for learning , practice and development.

Fully Distributed Mode (Real Cluster)

• In fully distributed mode, Hadoop runs on multiple machines forming a real cluster. There are one or more master node like Name Node, Resource Manager and also multiple worker nodes like Data Nodes, Node Managers.
  • It is used in production environments
  • There is high availability and fault tolerance
  • Replication factor usually set to 3

Comparison Table

Why Use Hadoop on AWS EC2?

• AWS EC2 gives us a cloud-based Linux server
• Hadoop needs a Linux environment
• We can practice real-world Big Data skills
• No need to buy our own hardware

Step 1: Launch an Ubuntu EC2 Instance

Provision an Ubuntu 22.04 EC2 instance (minimum t3.medium recommended with 20 GB EBS volume storage). Ensure Security Group ingress rules allow SSH (port 22) and HDFS Web UI (port 9870).

• Login your AWS management console and search and create EC2 instances
  • Name and Tag : wordcount-hadoop-server
  • AMI : Select Ubuntu
  • Instance Type : Select t3.medium
  • Key pair : Create Keypair [hadoopkey]
  • Security Group : Select SSH (22) , HTTP (80)
  • Storage: 20 GB SSD
• Then Launch Instances
• Connect it using SSH

ssh -i hadoopkey.pem ubuntu@ec2-public-ip
    

Step 2: Install Java

• Now, Install Java because Hadoop is written in java programming so we need java to run Hadoop.
• First update ubuntu system repository

sudo apt update
    

• Install open jdk

sudo apt install openjdk-8-jdk -y
    

• verify java version

java -version
    

Step 3: Download, Extract, and Export Environment Paths

• Download the stable Apache Hadoop binary distribution package into /usr/local:

wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xvzf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 /usr/local/hadoop
    

• Append environment variables into your user profile (~/.bashrc):

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
    

• Flush changes: source ~/.bashrc

• Configure Core Hadoop System XML Files
• Navigate to $HADOOP_HOME /etc/hadoop/ and update the XML configurations:
• Update core-site.xml (Defines default NameNode URI):

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
    

• Update hdfs-site.xml (Sets single-node replication factor)

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

• Update mapred-site.xml(Designates YARN as MapReduce framework):

mapreduce.framework.name
yarn
    

• Update yarn-site.xml Enables Shuffle service for MapReduce):

yarn.nodemanager.aux-services
mapreduce_shuffle
    

Step 4: Setup Passwordless SSH, Format NameNode, and Start Services

Configure passwordless local SSH loopback access so Hadoop daemons can start without prompting for credentials:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost
    

• Format the HDFS NameNode filesystem namespace

hdfs namenode -format

• Start HDFS and YARN background daemons:

start-dfs.sh
start-yarn.sh
jps # Verify running daemons: NameNode, DataNode, ResourceManager, NodeManager

Step 5: Upload Input Dataset & Run MapReduce Hadoop Word Count

Running the Hadoop Word Count job allows developers to verify if MapReduce framework and HDFS read/write functionalities are working properly across the single-node setup.

• Create a local sample text file (input.txt) with dummy words, then interact with HDFS.

# Create HDFS input directory structure
hdfs dfs -mkdir -p /user/ubuntu/input

# Upload local file to HDFS
hdfs dfs -put input.txt /user/ubuntu/input/

# Execute built-in MapReduce WordCount JAR file
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/ubuntu/input /user/ubuntu/output

# Inspect job output stored inside HDFS
hdfs dfs -cat /user/ubuntu/output/part-r-00000

Step 6: Explore dashboard using browser

• If you want explore dashboard in graphically then you have configure port in security group
  • Go to EC2 security Group
  • Click on edit Incound rule
  • Select custom TCP >> port (9870) >> anywwhere
  • and save it

Hadoop Infrastructure & Operational Alerts:

• Re-formatting NameNode Data Loss: Never execute hdfs namenode – format while Hadoop daemons are actively running. Formatting creates a new Cluster ID (clusterID) in the NameNode metadata directory. If DataNodes retain the old Cluster ID in their storage metadata blocks, the DataNodes will fail to handshake and crash automatically.

• AWS EC2 Public IP Web UI Access: To access the HDFS Web Console UI (htpp:// <EC2-Public-IP>:9870), verify that your EC2 Security Group contains a Custom TCP rule permitting inbound traffic on Port 9870.

Production Troubleshooting: Common Hadoop on EC2 Errors

Configuring single-node Hadoop clusters frequently triggers Java path mismatches or HDFS lock errors. Use the diagnostic steps below to resolve runtime failures

• Error 1: JAVA_HOME is not set Exception
  • The Error Log:

ERROR: JAVA_HOME is not set and could not be found.

• The Root Cause: Hadoop shell scripts (start-dfs.sh, hdfs) failed to inherit environment variables from ~/.bashrc during daemon initialization loops.

• The Fix: Explicitly declare the JAVA_HOME path inside $HADOOP_HOME /etc/hadoop/hadoop-env.sh . Open the file and insert your exact Java installation path:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

• Error 2: HDFS NameNode Locked in SafeMode
  • The Error Log:

org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/ubuntu/input. Name node is in safe mode.

• The Root Cause: Upon startup, NameNode enters SafeMode (read-only mode) to verify DataNode block reports. If free EBS disk space drops below 99% or memory constraints delay reports, NameNode stays locked in SafeMode.

• The Fix: Force NameNode to exit SafeMode manually by executing this HDFS administrative command:

hdfs dfsadmin -safemode leave

• Error 3: Output Directory Already Exists
  • The Error Log:

org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/ubuntu/output already exists

• The Root Cause: Hadoop MapReduce design explicitly prevents overwriting existing output paths to guard against accidental production data loss.

• The Fix: Delete the target HDFS output directory using the HDFS CLI before re-executing your MapReduce jar

hdfs dfs -rm -r /user/ubuntu/output