Archive for the ‘Hadoop’ Category

Hadoop 2.7 Installing on Ubuntu 14.04 (Pseudo-distributed mode)

Installing Java

Please read this artcile

Adding a dedicated Hadoop system user

For Hadoop, the accounts should have the same username on all of the nodes. This account is only for managing your Hadoop cluster. Once the
cluster daemons are up and running, you’ll be able to run your actual MapReduce jobs from other accounts.

bluething@ubuntu:~$ sudo addgroup hadoop
bluething@ubuntu:~$ sudo adduser --ingroup hadoop hadoop-user

Installing SSH

ssh has two main components:

ssh : The command we use to connect to remote machines – the client.
sshd : The daemon that is running on the server and allows clients to connect to the server.

The ssh is pre-enabled on Linux, but in order to start sshd daemon, we need to install ssh first.

Verify SSH installation

bluething@ubuntu:~$ which ssh
bluething@ubuntu:~$ which sshd
bluething@ubuntu:~$ which ssh-keygen

Install openssh-server

bluething@ubuntu:~$ sudo apt-get install openssh-server
bluething@ubuntu:~$ which sshd

Generate SSH key pair

bluething@ubuntu:~$ su - hadoop-user
hadoop-user@ubuntu:~$ ssh-keygen -t rsa -P ""
This command will create an RSA key pair with an empty password. Generally, using an empty password is not recommended, but in this case it is needed to unlock the key without your interaction (you don’t want to enter the passphrase every time Hadoop interacts with its nodes).
If you want to see public key form

hadoop-user@ubuntu:~$ more /home/hadoop-user/.ssh/id_rsa

Enable SSH access to your local machine with this newly created key.

hadoop-user@ubuntu:~$ cat $HOME/.ssh/ >> $HOME/.ssh/authorized_keys

Test the SSH setup by connecting to your local machine with the hadoop-user user. The step is also needed to save your local machine’s host key fingerprint to the hadoop-user user’s known_hosts file.

hadoop-user@ubuntu:~$ ssh localhost
The authenticity of host 'localhost (' can't be established.
ECDSA key fingerprint is 5b:5d:57:1a:08:34:51:9d:b2:26:3b:19:b3:84:eb:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

hadoop-user@ubuntu:~$ exit
Connection to localhost closed.


Installing Hadoop

Read more…

Categories: Hadoop