Configuring Apache Hadoop Cluster in a standalone machine

Introduction

           In this post I have tried to explain how to setup and configure Apache hadoop cluster with 2 or more nodes in a standalone machine probably ur windows laptop or desktop. This will help you to build map reduce program and run in a real cluster like environment and will help you to understand hadoop better.

           Apache hadoop is a free open source software release for reliable and scalable distributed computing. It is a framework that allows for distributed processing large data sets across clusters of computers.

During this hadoop cluster setup, at high level the following activities will be performed

v  Creating base nodes for the cluster

v  Setting up base operating system for the cluster

v  Setup hadoop dependencies in the nodes

v  Configure hadoop users ,access

v  Setup authenticity across the cluster nodes

v  Configure hadoop roles for the nodes

v  Run hadoop daemons for each roles

v  Browse for hadoop hdfs and job tracker sites

 

Creating base nodes for the cluster:

   If you are planning to try out this setup on your local windows laptop or desktop, download VMware player which is a free tool that can help you with setting up virtual machines with their local IP, so at the end you have a simple network of servers that can talk to each other. Nowadays laptop are coming with multiple cores and 4 GB of Memory, so it is easy to setup at least 3 nodes in your personal laptop or desktop.

 

Setup a Linux flavor of OS in the base nodes:

  On the base VM nodes you have set with VMware player, you can install a linux based OS with a ISO file, I choose ubuntu server as the OS, it is available free to download . Download the ISO and complete the VM creation with the VM Player.

  Once the OS installation is done, you will be ended with a root or sudo user for the server. You can get the IP address of the servers by typing the command ifconfig , note down the IP addresses for the servers.

 

Setup Hadoop and its dependencies:

  We have the servers setup with OS and a sudo user to operate on,now we can start setting up hadoop in the nodes.

  Apache hadoop has the following dependencies

1.       Java version 6 or higher

2.       SSH

    Download and set up in the server, I setup up JRE under a folder /opt/jre1.6.0_45 and set Java Home under ~/.bashrc , you can verify the setup by typing the command Java -version and check the version details displayed.

    SSH can be installed by using the command - sudo apt-get install openssh-server

    Verify SSH by executing the command SSH localhost to that machine itself.

    Download a stable version of hadoop . I choose 1.0.X as the version to setup.

    If you have downloaded the .tar.gz file you can use the command tar -zxvf {file.tar.gz} to unzip the contents. I have set it to the location /opt/hadoop-1.0.4 .

Configure Hadoop

We have hadoop and its dependencies set, we can now start configuring hadoop in that server, this involves the following activities

1.       Create a new user , say hadoop, In Ubuntu I used the command Adduser #user

2.       Add the sudo access to the user by editing /etc/sudoers file , this can be achieved by the following commands

a.      sudo visudo

add the line in the file hadoop ALL=(ALL:ALL) ALL

3.       Add full permission for this hadoop user to /opt/hadoop-1.0.4 where we have the hadoop binaries folder installed , this can be done by the following commands

a.      Chown –R hadoop:hadoop

b.      Chmod –R 777 hadoop-1.0.4

You have to repeat the above steps for all the nodes in the cluster or simply clone the virtual machines but make sure each virtual machine has got different IP Address. Consider you have created 3 nodes for this cluster.

Now we have 3 nodes created, we have to decide on the roles of the nodes considering one node to be master node playing roles of namenode and jobtracker and other nodes playing datanode and tasktracker, we can call the nodes as hdpMaster, hdpSlave1, hdpSlave2.

Configuring authenticated SSH access between master and other nodes

                We need to configure authenticated SSH access (password less) for hadoop user from masternode to rest of slavenodes. Perform the following steps to setup the same.

$ssh-keygen -t rsa ( generates the key file)

Copy the key file to all the slave machines

                $scp .ssh/id_rsa.pub hadoop@192.168.8.129:~hadoop/.ssh/authorized_keys (Slave1)

                $scp .ssh/id_rsa.pub hadoop@192.168.8.130:~hadoop/.ssh/authorized_keys  (Slave2)

                You should also able to ssh without password into the same, otherwise you have to do the following to do the same.

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

                Once the key is added to authorized keys of master, password less access to machines will be possible.

               Verify whether you are able to connect using ssh to localhost and all the slaves by using ssh command

                ssh localhost

                ssh slave1IP

                ssh slave2IP

 

HostEntry for the Server:

         Update the host file with hostnames at etc/hosts , if you want to call the servers with hostnames

Configure hadoop roles for master and slaves:

         We have all set for the hadoop to start, we are at the last step of configuring the roles for the nodes and start the cluster.

         In the master node, perform the following steps

1.       Go to the HadoopHome \ Conf location

2.       Update hadoop-env.sh with JAVA_HOME location to the Java installation path

3.       Update core-site.xml to the following

4.       Update hdfs-site.xml to the following

5.       Update mapred-site.xml to the following

6.       Update masters file with the masterhostname

7.       Update slaves file with all slavehostname.

Repeat step 1 -4 to all the slave nodes.

Hadoop cluster is now configured for hdfs and mapreduce. We can start the corresponding daemons on the cluster

Step 1 : go to HadoopHome location

Step 2: Format namenode by running the command bin/hadoop namenode –format

Step 3: go to bin folder, Run namenode, datanode daemons , Run Jobtracker, tasktracker daemons

Option 1: Run ./start-all.sh in master node, this will start all the daemons in all the nodes cluster as configured in masters,slaves file

Option 2: Run ./start-dfs.sh in master node, this will start namenode and datanodes , Run ./start-mapred.sh , this will start jobtracker and tasktracker in the nodes.

Option 3:Run the following

In Master node

 ./hadoop-daemon.sh start namenode

./hadoop-daemon.sh start jobtracker

In Slaves node run

./hadoop-daemon.sh start datanode

./hadoop-daemon.sh start tasktracker

You can check the logs of the nodes or any errors during initialization under HadoopHome/logs in each of the nodes.

If everything went fine, you should be able to see the following sites for tracking hdfs and hadoop jobs

http://masternode:50070/dfshealth.jsp - to track hdfs and its health

http://masternode:50030/jobtracker.jsp - to track job running and its status

 

Reference : Apache hadoop cluster setup

A2A 'Cloud Comparison' - Database as a Service

This is part of my series of articles on A2A Cloud Comparison. In my previous articles I was explaining my views of A2A Comparison with Compute and Storage. In this article I will provide my views on Database as a Service with Amazon and Azure.

Introduction
We all know how data is crucial to an application take an example whether it is a banking application or an online music store application, data is very important to the whole system. Say you have recently registered and created a user on a specific site and if the user identity is not found next time when you login to the site think how much hesitation will get and you will think twice before continuing to use the site. Think of what will happen if you lose some data in a critical financial application. Losing the data will incur heavy loss to the system or make the application really obsolete. The reason why I am talking about data criticality is because in this blog I am going to talk about the database as a service offering from the cloud computing providers.
When we talk about data most of the applications store their data in a database and managing the database will be a crucial task for the system. Database administration helps to manage the database and assures to keep the database updated and highly available. I want to list some to tasks performed as part of database administration
1. Patching the database software up to date
2. Taking backups of the database
3. Maintaining the backup for the specified retention period
4. Point in time recovery

Database as Service
What if all the database administration tasks have been taken care and have ability to scale the capacity with high availability and reliability? Database as a Service is the answer for that.

Amazon and Azure Offerings
Both Amazon and Azure provides offerings in the database as a service space and are differentiated in some ways. Amazon provides its offering as RDS (Relational Database as a Service) while Microsoft Azure provides its offering as SQL Azure.
Amazon operates RDS in Infrastructure as a Service space while Microsoft SQL Azure operates at Platform as a Service space, I will be explaining it in detail below. Following the general cloud pricing model this service will also be charged in a Pay as you use model.

RDS:
Amazon offering for Database as a service called RDS (Relational Database as a Service) provides database service for MYSQL database. Recently Amazon has made an announcement that will extend RDS for Oracle database, that means you will be able to create an Oracle database with all the setup ready in matter of minutes and you can able to create and delete the instances with hourly chargeback model and with all database administration tasks taken care..Sounds interesting?
Every RDS instance in Amazon will get a dedicated virtual server instance, database storages with all the data backup and retention policies configured, this is why I called RDS operating in Infrastructure as a Service space and because of its underlying virtualization model the instance can be migrated to a bigger server configuration if needed. Database servers can also be configured for Read replication or Multi Availability Zone deployment for high availability and Disaster Recovery.
Recently I have to validate the performance of Oracle database in a specific use case for a POC, for scenarios like this it will be difficult in non cloud model because Oracle software licenses will be charged for duration of a year at least and the licenses are Processor based or Socket based. It will be difficult to compromise with express edition or a single socket license as we have to validate performance scenario and now with cloud model it is easy to execute, create and use it for the period needed and release it when POC is done , as simple as that.

SQL Azure:
Microsoft offering for Database as a Service called SQL Azure provides service for SQL Server database. With SQL Azure we will be able to create databases for 1GB, 5GB up to a maximum of 50GB. We can create a smaller DB during creation and can later alter to a maximum of 50GB with all the database management tasks taken care operating in a pay as you use model.
Microsoft operates SQL Azure in a way bit different from Amazon RDS. Unlike RDS SQL Azure does not spare a dedicated virtual server for databases instead multiple SQL Azure databases will be hosted in a bigger SQL Server instance and will be operating more like a shared multi tenant environment with all the tenant specific security measures taken care, this architecture will be abstracted from the end user as the end user will be able to operate the database in a usual way and the user is assured with high availability and scalability.
One thing that has to take care in SQL Azure is that it can scale to a maximum of 50GB as of now and beyond that we have to plan for horizontal scaling of database in our application architecture.

References:
http://social.technet.microsoft.com/wiki/contents/articles/inside-sql-azure.aspx
http://aws.amazon.com/rds/

A2A ‘Cloud Comparison’ – Storage Services

This is part of my series of article 'A2A Cloud Comparison' ; in my previous article I have compared Amazon and Azure on Computing Services space. In this article I have given my view on Cloud Storage Services in general and the corresponding services by Amazon and Azure Cloud Providers.
Storage in Cloud
One of the important services that are provided by Cloud is the Storage Service. Cloud Storage provides enormous amount of storage space that is accessible over internet with features added on top of it. Also as with other cloud services this comes with Pay as you use model. Let us understand why the storage services in cloud is going to be important, year by year the cost of storage disks keep on reducing but still the enterprise storage cost keep on increasing year by year, the problem with conventional storage costing is that even though the hardware cost keeps on reducing cost on operation and maintenance keeps the total cost increased, also it is difficult to keep with the exponential need in the storage needs. Cloud Storage Services tries to address all these problems.

Understanding Storage in Cloud:
Cloud Storage operates on a base concept called Storage Virtualization. Storage Virtualization system provides a logical data store that maps over the physical storage system through a mapping table.
Storage Virtualization in general achieves the following
1. Location independence – Abstracts the physical location and thus enables data movement across different physical locations.
2. Replication – Enables replication of the storage data across multiple locations
3. Data migration – Enables movement of storage data to a faster / better infrastructure if needed.
4. Dynamic scaling - Enables to scale the capacity of the storage space when needed

Storage Services in Amazon and Azure
Amazon, Azure the top public cloud computing service providers provides services in Storage segment. Both of them provide similar type of services in storage segment. These storage services can be accessed by a REST based API or web service API calls.



Let us try to compare the cost of these storage services by these vendors. Generally the cost of these services will vary based on geographic location and also will be revised (generally reduced), the costing I am mentioning is as of today.


(Please note that billing fees are subject to change.)

Please refer to the following links for the detailed pricing
http://aws.amazon.com/s3/#pricing
http://www.microsoft.com/windowsazure/pricing/

Security options
Security in Data Transition:
Security in data transition can be achieved by means of secured http channel.
Security in Data Source:
Highly sensitive data that needs to be secured at the source can be achieved by means of data encryptions.
Security in Access:
Cloud providers are coming up with Authentication, Authorization mechanism by which access to these resources can be secured.
Security in Virtualization:
Virtual Servers in the same physical servers are properly secured by means of virtual firewall by the cloud providers and hence data is kept secured between virtual servers on same physical server.

Best Practices
1. Choose the Cloud Storage Data centre location closer to the end user
2. Segregate the data into different buckets(Amazon) or Containers(Azure) so that different level of security access can be achieved
3. Partition the data properly to achieve higher throughput and efficiency.

CDN Integration
Both Amazon and Azure provides Content Delivery Network (CDN) that can be integrated with their storage services to provide closer delivery of data to the clients with higher performance and better reliability.

Tools
There are few cloud storage explorer management tools that are available that facilitates a user to view the data on cloud storage
Cloudberry Explorer - http://cloudberrylab.com/
Explorer Tools: S3Fox, BucketExplorer, awszone.com
Azure Storage Explorer - http://www.cerebrata.com/Blog/file.axd?file=2009%2F10%2Fcomparing_azure_storage_management_tools.pdf
Azure Storage Manager - http://azurestoragemanager.codeplex.com/

Other Cloud Storage Providers in the market:
Nirvanix - http://www.nirvanix.com/
EMS Automos - http://www.atmosonline.com/

A2A ‘Cloud Comparison’ – Compute Service

A2A ‘Cloud Comparison’ – Compute Service

As many of us know Amazon and Azure are among the major providers in the public cloud service space. This will be series of blogs depicting my views on comparing Amazon to Azure (A 2 A) on public cloud services on various dimensions of their services like Compute, Storage, Bandwidth, Pricing, Security, DB Services, and CDN etc. Please tag to this space to follow closely on the series.
In this current blog I have taken compute service offering from both of these providers and provided their features as per my knowledge.

Amazon EC2 Compute Instances:
Amazon provides services in Infrastructure services space where in compute instances it provides compute services in terms of virtual servers, the compute instances so called EC2 (Elastic Compute) instances provide different flavours in terms of hardware configuration and software configuration, some of the flavours in hardware are Micro, Small, Large, XLarge, High CPU, High Memory etc., you can find more details of it here at http://aws.amazon.com/ec2/instance-types/ , the costing of the instances varies based on the flavour. Each instance flavour can differ in terms of hardware configuration and software configuration. Amazon as a provider provides instance for some predefined software like Windows Server 2003, 2008, SQL Server editions, RHEL etc. In addition amazon has partnership with major vendors like IBM, Oracle and provides pre built alliances, for example you can have a prebuilt appliance with oracle 11g with different hardware configuration provided by Oracle, and similarly for IBM you have appliances provided by them.
Details of the partnership EC2 instances for IBM and Oracle are available under
http://aws.amazon.com/solutions/global-solution-providers/oracle/
http://aws.amazon.com/solutions/global-solution-providers/ibm/

Some of the benefits you can find with Amazon Compute instances are
1. Prebuilt appliance and save your time and avoid expertise from setting up with proper environment
2. Some of the instances through partnership comes as Pay as you use model and hence avoids licensing and costing issues, suppose you want to test or do some POC with IBM Web sphere Portal Server for a week or even a day you can very well find the instance and use it with amazon ec2 instance in no matter of time.
3. Many software vendors started providing their products through Amazon instances with the correct environment set, this way it becomes easy for customers to try out any software of their interest with less turnaround time.
4. Start and terminate instances when ever needed and pay only for used time.
5. Set firewall and other security for the instances as you need.
6. Ability to monitor the health status of the instances.
7. Easy to migrate existing applications with same flavour on the cloud platform.

Azure Compute instances:
Microsoft Azure as we know operates in Platform Services layer, in the sense user won’t be exposed to the server directly, but when it hosts the application it provides a virtual server for running the application. Similar to Amazon Azure also provides some option on the virtual server configuration like Small, Medium, Large, Extra Large etc. Details of the instance can be found at http://www.microsoft.com/windowsazure/windowsazure/default.aspx With respect to operational model Azure provides compute instances in two different flavours as web role and worker role. Web role instances are used when the applications needs front end handlers handled by IIS web server and worker roles are used when the application needs a back end handling process ex: a batch job application or a windows service application.

Benefits of Azure Compute Instances:
1. Instances are self health monitored by Azure Fabric
2. Auto scaling can be enabled on the instances.
3. Control security policy over the instance.
4. Easy to build and migrate applications based on IIS7 and ASP.Net
5. Development fabric on Windows Azure SDK provides a simulated environment for service deployments and role instances on local machine.

To make a comparison study on Amazon and Azure with respect to compute instances
1. Amazon Compute instances are at infrastructure level and hence have more control over the instances, while for Azure Compute the control is limited as it provides platform services. In Azure some of the overhead like application monitoring , high availability it taken care by Azure fabric where in Amazon we have integrate few Amazon services to achieve it manually.
2. Azure allows deploying only one role per compute instance, where in Amazon you can deploy multiple applications / services as we do with normal servers. For example if you want to deploy a ASP.Net based application and WCF service at back end, you may need 2 compute instance in Azure, where in Amazon EC2 instances you can deploy them in same virtual machine.
3. Azure instances are self monitored and controlled by Fabric where in Amazon EC2 instances we add a service called cloud watch to monitor specific instances.
4. Azure instances will allow running applications based on Windows environment where in Amazon we can run applications based on windows and Linux environments.
5. Applications with some 3rd party dependencies or using commercial of the shelf products will be less suitable to migrate to Azure as the platform needs the dependencies on the Azure platform and licensing of the products needs to be worked out.
Microsoft is planning to release virtual servers (VMRole) in Infrastructure as a Service space similar to Amazon in near future http://blogs.msdn.com/b/usisvde/archive/2010/03/29/vm-support-in-windows-azure.aspx , with vmrole Azure platform will gain more power and benefits for migrating Microsoft based applications to cloud.

Cloud Hosting versus Web Hosting

I have talked about cloud services interms of infrastructure services, platform services, software services , when detailing about infrastructure services many people have this doubt in mind , how is the cloud infrastructure services ( ex : Amazon ) differ from the normal web hosting providers,if I use virtual servers from web hosting providers, how cloud hosting is different ?, i thought to provide my thoughts on that.

Let us talk a sometime to talk about web hosting providers, there are different type of providers, with dedicated hosting provider we can rent physical servers not shared with anybody and have full control over the server including the server administration, with shared web hosting provider we can share the server space with others to be cost efficient but have less control over the server like administration , we can also procure interms of virtual servers and use.

So what's the big deal with cloud infrastructure providers like amazon , they also provides servers in the form of virtual servers so what's the big difference and what makes amazon to be called as cloud provider?

When we say cloud provider the main difference we have to clearly observe is the utility model of computing in all dimensions of usage , the chargeback will happen based on how much we utilize the cloud elements like compute units, bandwidth, power usage, storage etc..Also the turn around time to set up an infrastructure with amazon will be much easier and quick compared to web hosting providers.

For example when a rent a server with a hosting provider we have to commit for their space say interms of months / years. It is not easy to dynamically expand and reduce the space depends on our need, you need to have minium commitment for specific duration. Also you have to get dedicated intenet bandwidth for our application need. Similarly we have to procure storage requirements for our need, all these needs minimum commitment with the provider. We cannot dynamically scale up and down instantly with general web hosting providers.

Taking an example with Amazon Infrastructure Cloud hosting provider we see how the cloud elements can be used in utilization model. Amazon provides a simple web interface in the form of plugin with firefox called 'Elastic fox' with which any user having an amazon account can securely create , destroy instances , attach, detach disk , set security settings for the instances , it also exposes SDK to operate on the elements, so you can programatically operate on the elements based on your application needs. So you basically pay for what you use and scale dynamically for seasonal needs. It also assures of high availability .

In addition to these, amazon infrastructure services provides blob storage services, Simple DB services, Amazon Relational database services, simple queue services , notification services. Using all these services in utilization model you can effectively build an architecture that use these infrastructure in a effective way to operate your app in OPEX (Operational Expense )model than in CAPEX ( Capital Expense ) model.

Cloud platform services like Microsoft Azure , Google App engine provides more services built on top of infrastructure services and cloud software service providers like sales force provides services at higher level than platform services.

Getting the power of cloud computing ...

SaaS(Software as a Service) versus Cloud

Many of us will have this doubt in mind , what is a SaaS(Software as a Service) application?, what is cloud application? Can I say all SaaS application are cloud applications? , or the vice versa is true , how both of these applications are interrelated..
Does all cloud applications provide SaaS type of service? When I develop an application say in Windows Azure or Amazon EC2, will I get the appliation in SaaS model? The answer is big No.
When we define cloud computing we say SaaS as one of the services of cloud computing , what does it mean then? Let us try to understand what is SaaS ? SaaS is Software as a Service in which the application is available as a service , where in a new customer wants to use that application for his usage, he can just pay and on board as a tenant to the SaaS application , do some level of customization available and use that for his use with the specified level of data security and isolation needed, so how this SaaS type of application is related to cloud computing?
Designing SaaS type of application is comparitivily difficult to design and implement because of its extensive functionalities. High availability and massive scalability are some of the basic requirements of SaaS type of applications and cloud computing techiques helps to solve the high availability and scalability in a simple way.
To say in a simple way, cloud computing enables to build SaaS applications easily , SaaS enablement is achieved easily through cloud computing techniques.

Cloud computing segments

cloud computing solutions can be classfied into three broad segments, solutions on private cloud or public cloud or hybrid cloud.

Enterprise customers who already has datacenters in place that will prefer to migrate their datacenters into private cloud , also companies that faces security as top most concern and takes no excuses unless they find any proven record on using public cloud , they will prefer to take their path towards private cloud solutions.

Companies that operates on huge amount of data and do manipulations on those data on temporary and permanent basis and want to share the data among business will prefer to go for utilizing public cloud storage and computing. For example media and entertainment companies will prefer to move towards public cloud where they can store and share huge data in widely spread public cloud.

Companies that has some medium level data centers and want to extend to public cloud on need and for less critical applications will prefer to migrate to build hybrid solutions.. for examples corporates can build a medium level data center for high critical applications and use public cloud for less critical applications.. that way they can manage concerns on security as well as costs.

In the fore coming I will talk about players in private cloud ,public cloud etc..