What Distribution of Hadoop Do I Use? 100% Apache Hadoop or Something Else?

Do we want to choose a Hadoop distribution based on the Apache source or do we go with a distribution that has improved on or modified the Hadoop architecture? 

            There are some great Big Data distributions out there that have made their own improvements on the Hadoop architecture, like MapR. MapR has thrown out the Hadoop file system (HDFS) and replaced it with their own file system which they call MapR-FS. By rewriting the file system, MapR has done away with the need for the Namenode and Secondary Namenode services. MapR claims that their proprietary file system is much faster that HDFS and that MapR's Apache Hadoop distribution provides full data protection, no single points of failure, improved performance, and dramatic ease of use advantages. MapR provides three versions of their product known as M3, M5 and M7. M3 is a free version of the M5 product with degraded availability features. M7 is like M5, but adds a purpose built rewrite of HBase that implements the HBase API directly in the file-system layer.

                Another distribution which keeps HDFS but changes how Namenode works is WANdisco. WANdisco uses three Namenodes to provide High Availability (HA). In WANdisco's implementation, they remove the need for a Secondary Namenode, JobTracker, or TaskTracker. WANdisco calls this their Non-Stop NameNode which uses it's patented replication technology to turns the NameNode into an active-active shared-nothing cluster that delivers optimum performance, scalability and availability on a 24-by-7 basis without any downtime or data loss.

                Both Hortonworks  and Cloudera use a 100% Open Source Apache Hadoop Distribution. Cloudera was founded in 2009 by Christophe Bisciglia from Google, Amr Awadallah from Yahoo, and Jeff Hammerbacher from Facebook. Cloudera announced this year the release of Impala. Impala is an open-source distributed query engine for Apache Hadoop. Cloudera is a sponsor of the Apache Software Foundation.

                Hortonworks was formed in 2011 by Yahoo and Benchmark Capital. Hortonwork has a great community involvement. Hortonworks, along with Yahoo, sponsor the Hadoop Summit. Hadoop Summit is an Apache Hadoop community event that outlines the evolution of Apache Hadoop into the next-generation enterprise data platform and features presentations from community developers, experienced users and administrators, as well as a vast array of ecosystem solution providers. And like Cloudera, Hortonworks is a sponsor of the Apache Software Foundation. In response to Cloudera's Impala release, Hortonworks announced its' Stinger Initiative. Stinger represents a concerted effort by Hortonworks and the broader Apache community to improve Hive performance and better serve business intelligence use cases such as interactive data exploration, visualization and parameterized reporting.

                So to answer the first question I think we need to ask what may be the most important question; what feature is most important in our implementation of Hadoop? For my company it's support. Why support? Because we want to stand up a small cluster, 6 datanodes with 100TB of storage  in our local datacenter to improve the time it takes us to load and query large static, transactional type data, which we currently do in SQLSerever. Having easy access to technical support, blogs, how to's, mailing list, knowledge base articles, and white papers will ensure the success of our Hadoop implementation. With that said, we can now ask the original question; do we want to choose a Hadoop distribution based on the Apache source or do we go with a distribution that has improved on or modified the Hadoop architecture?

                I believe for us the answer is to use a 100% Open Source Apache Hadoop Distribution. We need to have as many resources available to us so that we can learn and we want a solution that is fully backed up by the vast Apache Hadoop community. By using Apache Hadoop, we can be assured that bugs will be fixed much faster, new features will be added and will be available in faster release cycles, and we will have a larger support base.

Hadoop As An Alternative To SQL Server?

It's been a very long time since my last blog post. I am still very much involved in SQL Server, but more accurately, involved in managing large volumes of data. Not as large as some, but larger than many. Just in the month of October, my company managed to upload 20TB in SQL Server. All static transnational data from other companies that we will run analysis on to answer questions about that data. So how do we do this currently? Well, most of our servers are virtual servers. 3PAR tier 1 fiber channel storage. Tables with row counts approaching a billion rows and greater are partitioned. We work with our analyst to improve query performance as they work to analyze the data. We are often under pressure to provide our analysis to customers in a short amount of time, therefore, we do not have much time to stage the data. Also, because the data is static, we have no use for transaction logs, which are big drain on performance. If only there was an option to turn off transaction logging.

And so here is where I find myself today. Hadoop. Could Hadoop help us? About a year ago I started reading about Hadoop and attending Hadoop and other NoSQL meet-ups. Over that time, I have become convinced that it can. I needed to set up a test environment, so I followed Michael Nolls' Hadoop tutorial, which can be found here; http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

I installed Hadoop on a single node, not without overcoming an obvious learning curve as it has been a very long time since I used Linux. I need to add that it's helpful to know a little Linux. After successfully installing Hadoop on a single node, I moved on to Michael's tutorial for installing Hadoop in a cluster. My test environment uses 4 nodes. I learned that the Hadoop logs are the key for correcting errors. And even though I had successfully installed Hadoop on my cluster of Dell desktop boxes, I still needed to learn how I might duplicate the analysis that we run in SQL Server in Hadoop.

Now, Hadoop has many data warehouse and database infrastructures that have been developed to sit on top of Hadoop. Some of these are Hive, Hbase, Pig, and Accumulo, just to name a few. But which one is best for our test case? Next week I will be driving up to Philly to attend Hadoop developer training provided by Hortonworks. It is a 4 day course designed for developers who want to better understand how to create Apache Hadoop solutions. The course consists of 50% hands-on lab exercises and 50% presentation material. It's very exciting to learn new technologies and I guess that's why I love what I do. Hopefully, I'll find answers to my many questions and post a detailed blog post of how I translated our processes that we currently perform in SQL Server, to our Hadoop test cluster. So stay tuned!

Free ebook: Introducing Microsoft SQL Server 2012

Microsoft Press is releasing a free ebook: Introducing Microsoft SQL Server 2012 (second DRAFT preview)by Ross Mistry and Stacia Misner. This Draft Preview release is only available in PDF format. To get the full details, look here; Microsoft Press.

Correcting Hadoop's HDFS java.io.IOException Errors

This past Christmas and New Year, like the last three years now, I accompanied my wife to Bogota, Colombia for the holidays. The difference about this trip was; our pet Mickey was going with us, we had no side trips planned, and I was taking my new Lenovo T420 with me. We arrived in Bogota late at night on December, 22nd, and after being greeted by family, we took a cab to my mother-in-laws house. There’s something both exciting and terrifying about cab rides in Bogota, but after time, for me at least, it’s just fun.

I woke up the next morning well rested and we began our Christmas vacation. I began setting up a new wireless router that I had brought with me so that I could work remotely from wherever was most comfortable. After having installed the router, I connected with my new laptop and set my priorities for the task that I needed to complete. Was the typical task, update my time, respond to some emails, finalize peer reviews, and check up on database backups. After these tasks were completed, I had time to relax.

What was planned for us was day to day, but mostly we would go out to have lunch or dinner with friends and family, then return home. On a few nights, we engaged in consuming heavy amounts of adult beverages and dancing which is the custom in Colombia. Most of our time however, was spent at home. I would wake up early, before anyone else, and sit at the dining room table, next to the window with my laptop and watch the sun rise over the mountains enjoying some fresh Colombian coffee.

With nothing to do so early in the morning, I decided to log onto my desktop back in DC. When I connected to my desktop, I saw that I had left open a ssh connection to a small Hadoop cluster I had setup to do some testing, except HDFS was not working properly. This was the perfect opportunity to find out what went wrong in my install and configuration of the cluster. The only catch was, is that I would have do everything from the command shell, no gui. I had followed Michael Noll’s “Running Hadoop on Ubuntu”,but now, I was getting errors in the namenode logs.

I was getting java.io.IOException errors. In Michael Noll’s how-to, he describes how he addressed this error by reformatting the cluster. He described how he stopped all running daemons and deleted the /app/hadoop/tmp/hdf/name/data directory and then ran bin/hadoop namenode –format . Somewhere in my troubleshooting my errors and researching online, I found that it was also a good idea to add the following properties to the hdfs-site.xml configuration file.

< !-- Adding dfs.data.dir dfs.name.dir 1/1/2012. -- >
< property >
< name > dfs.data.dir < /name >
< value > /app/hadoop/tmp/dfs/name/data < /value >
< final > true < /final >
< /property >
< property >
< name > dfs.name.dir < /name >
< value > /app/hadoop/tmp/dfs/name < /value >
< final > true < /final >
< /property >

Also, if you get permission denied (publickey,password), you may want to check that the paths for the properties you added to the hdfs-site.xml file are correct. If this problem persist, you might try running the following on all nodes;

sudo chown –R hduser:hadoop /app/hadoop

Some of the other errors that I ran into were as follows;

Cannot lock storage /app/hadoop/tmp/dfs/name. The directory is already locked.
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All storage directories are inaccessible.
ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException: Problem binding to Address already in use

I did not do a good job at keeping notes during my troubleshooting these issues, but it seemed that whenever I would try to fix one thing, a different error would pop up. I did find that I had made an typo in the hdfs-site.xml file. At the end of each path, I had added a /. Therefore, instead of /app/hadoop/tmp/dfs/name, I had /app/hadoop/tmp/dfs/name/. But once I corrected that and delete all data in the HDFS directory and then ran format, everything worked! So here is how that went.

After stopping all daemons and correcting the paths in the hdfs-site.xml file, I then deleted all data in the HDFS directory on all nodes.

hduser@bigdata1:/app/hadoop/tmp/dfs$">hduser@bigdata1:/app/hadoop/tmp/dfs$ sudo rm -rf *

Then, I ran the format.

hduser@bigdata1:/usr/local/hadoop/hadoop$ bin/hadoop namenode –format

The output looks like;

12/01/04 07:59:15 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = bigdata1/172.20.10.92
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
12/01/04 07:59:15 INFO namenode.FSNamesystem: fsOwner=hduser,hadoop,adm,dialout,fax,cdrom,floppy,tape,audio,dip,video,plugdev,fuse,lpadmin,netdev,admin,sambashare
12/01/04 07:59:15 INFO namenode.FSNamesystem: supergroup=supergroup
12/01/04 07:59:15 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/01/04 07:59:15 INFO common.Storage: Image file of size 96 saved in 0 seconds.
12/01/04 07:59:15 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
12/01/04 07:59:15 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bigdata1/172.20.10.92
************************************************************/

Next, I started HDFS.

hduser@bigdata1:/usr/local/hadoop/hadoop$ bin/start-dfs.sh

It’s output was;

starting namenode, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-namenode-bigdata1.out
bigdata2: starting datanode, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-datanode-bigdata2.out
bigdata3: starting datanode, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-datanode-bigdata3.out
bigdata4: starting datanode, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-datanode-bigdata4.out
bigdata1: starting secondarynamenode, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-secondarynamenode-bigdata1.out

I ran the HDFS Admin Report to see the status of my cluster.

hduser@bigdata1:/usr/local/hadoop/hadoop$ bin/hadoop dfsadmin –report

The report displays the following;

Configured Capacity: 206701436928 (192.51 GB)
Present Capacity: 186873368576 (174.04 GB)
DFS Remaining: 186873294848 (174.04 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Name: 172.20.10.127:50010
Decommission Status : Normal
Configured Capacity: 68900478976 (64.17 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6721228800 (6.26 GB)
DFS Remaining: 62179225600(57.91 GB)
DFS Used%: 0%
DFS Remaining%: 90.24%
Last contact: Wed Jan 04 08:00:20 EST 2012


Name: 172.20.10.128:50010
Decommission Status : Normal
Configured Capacity: 68900478976 (64.17 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6732939264 (6.27 GB)
DFS Remaining: 62167515136(57.9 GB)
DFS Used%: 0%
DFS Remaining%: 90.23%
Last contact: Wed Jan 04 08:00:20 EST 2012


Name: 172.20.10.48:50010
Decommission Status : Normal
Configured Capacity: 68900478976 (64.17 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6373900288 (5.94 GB)
DFS Remaining: 62526554112(58.23 GB)
DFS Used%: 0%
DFS Remaining%: 90.75%
Last contact: Wed Jan 04 08:00:17 EST 2012

After this I started MapReduce.

hduser@bigdata1:/usr/local/hadoop/hadoop$ bin/start-mapred.sh

It’s output was this;

starting jobtracker, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-jobtracker-bigdata1.out
bigdata3: starting tasktracker, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-tasktracker-bigdata3.out
bigdata2: starting tasktracker, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-tasktracker-bigdata2.out
bigdata4: starting tasktracker, logging to /usr/local/hadoop/hadoop/bin/../logs/hadoop-hduser-tasktracker-bigdata4.out

Success!

Now there are three web interface URLs that you can use to check up on your clusters health, there are;

http://localhost:50030/ – web UI for MapReduce job tracker(s)
http://localhost:50060/ – web UI for task tracker(s)
http://localhost:50070/ – web UI for HDFS name node(s)

It's time again for SQL Saturday Washington DC! With last year's event being such a huge success, this year's event will be even better! We are expecting over 250 attendees to show up for a full day of training from some of the best speakers in the SQL Server community.

This is one event you don't want to miss! Everyone who attends will get a SQL Saturday T-Shirt and lots of swag! Free breakfast and lunch will be provided. We will have several SQL Server vendors that will be presenting the latest upgrades and solutions. And at the end of the day, we will be raffling off some awesome door prizes provided by our many sponsors!

SQL Saturday is a FREE one day training events for SQL Server professionals. SQL Saturday was initially the idea of three DBAs that wanted a Code Camp style event just for SQL Server professionals. It began with the first SQL Saturday in Tampa, Florida. The event was such a huge success, more events followed. After over 40 successful events, The Professional Association for SQL Server took over the administration of SQL Saturday events.

SQL Saturday events are usually divided into three tracks consisting of BI, Database Development, and Database Administration. Typically there will be five sessions per track. SQL Saturday tries to recruit local SQL Server professionals to present at the event, but occasionally, more nationally known speakers may also present.

SQL Saturday is all about sharing issues and solutions, and gaining knowledge that will make you a better SQL Server professional.

We’ll see you there!!!

Register at: http://www.sqlsaturday.com/96/eventhome.aspx

FOLLOW US!

SQL Server: An error occurred while executing batch. Error message is: The directory name is invalid.

I logged onto the server this morning to delete some files because the disk had run out of free space. After deleting some files, I went to run some maintenance scripts in SSMS and received the following error:

An error occurred while executing batch. Error message is: The directory name is invalid.

After doing some online research, I found out that the reason for this error is because SQL Server cannot find the temp folder in which to store the query results. To correct this, try logging off and back on to the machine that you are running SSMS on. If the error persists, reboot the machine.

Cannot edit job steps in SSMS 2008 R2

When attempting edit the job step, or view the job step details, I received the following error:

Creating an instance of the COM component with CLSID {AA40D1D6-CAEF-4A56-B9BB-D0D3DC976BA2} from the IClassFactory failed due to the following error: c001f011. (Microsoft.SqlServer.ManagedDTS)

You may also get this error when attempting to create a new job step.

I did a search and found several old post requesting help with the same problem. Microsoft corrected the problem with the last Hotfix; Cumulative Update package 7 for SQL Server 2008 R2.

For more information on Cumulative Update package 7 for SQL Server 2008 R2 and how to download it, go here; http://support.microsoft.com/kb/2507770