Wednesday, July 10, 2013

Speed up hadoop development with progressive testing

Debugging Hadoop jobs can be a huge pain.  The cycle time is slow, and error messages are often uninformative --- especially if you're using Hadoop streaming, or working on EMR.

I once found myself trying to debug a job that took a full six hours to fail.  It took more than a week -- a whole week! -- to find and fix the problem.  Of course, I was doing other things at the same time, but the need to constantly check up on the status of the job was a huge drain on my energy and productivity.  It was a Very Bad Week.



Painful experiences like this have taught me to follow a test-driven approach to hadoop development.  Whenever I'm working on a new hadoop-based data pipe, my goal is to isolate six distinct kinds of problems that arise in hadoop development.

  1. Explore the data: The pipe must accept data from a given format, which might not be fully understood at the outset.
  2. Test basic logic: The pipe must execute the intended data transformation for "normal" data. 
  3. Test edge cases: The pipe must deal gracefully with edge cases, missing or misformatted fields, rare divide-by-zeroes, etc. 
  4. Test deployment parameters: The pipe must be deployable on hadoop, with all the right filenames, code dependencies, and permissions.
  5. Test cluster performance: For big enough jobs, the pipe must run efficiently.  If not, we need to tune or scale up the cluster.
  6. Test scheduling parameters: Once pipes are built, routine jobs must be scheduled and executed.

Each of these steps requires different test data and different methods for trapping and diagnosing errors.  Therefore, the goal is to make sure to (1) tackle problems one at a time, and (2) solve each kind of problem in the environment with the fastest cycle time.

Steps 1 through 3 should be solved locally, using progressively larger data sets.  Steps 4 and 5 must be run remotely, again using progressively larger data sets.

Step 6 depends on your scheduling system and has a very slow cycle time (i.e. you must wait a day to test whether your daily jobs run on the proper schedule.).  However, it's independent of hadoop, so you can build, test, and deploy it separately.  (There may be some crossover with #4, but you can test this with small data sets.)

Going through six different rounds of testing may seem like overkill, but in my experience it's absolutely worth it.  Very likely, you'll encounter at least one new bug/mistake/unanticipated case at each stage.  Progressive testing ensures that each bug is dealt with as quickly as possible, and prevents them from ganging up on you.

Other suggestions:
  • Definitely use an abstraction layer that allows you to seamlessly deploy local code to your staging and production clusters.  Cascalog and mrJob are good examples.  Otherwise, you'll find yourself solving steps 2 and 3 all over again in deployment.
  • Config files and object-oriented code can reduce a lot of headaches in step 4.  Most of your deployment hooks can be written once and saved in a config file.  If you have strong naming conventions, then most of your filenames can be constructed (and tested) programmatically.  It's amazing how many hours you can waste debugging a simple typo in hadoop.  Good OOP will spare you many of these headaches.
  • Part of the beauty of Hive and HBase is that they abstract away most of the potential pitfalls on the deployment side, especially in step 4.  By the same token, tools like Azkaban and Oozie can take a lot of the pain out of step 6.  (Be careful, though -- each of these scheduling tools has its limitations.)

34 comments:

  1. Hadoop keeps track of where the data resides ,there are multiple copy stores, data stored on a server.
    Hadoop Development

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. This website is very helpful for the students who need info about the Hadoop courses.i appreciate for your post. thanks for shearing it with us. keep it up.
    Hadoop Training in hyderabad

    ReplyDelete
  4. Hadoop Developer Online Training, ONLINE TRAINING – IT SUPPORT – CORPORATE TRAINING http://www.21cssindia.com/courses/hadoop-online-training-182.html The 21st Century Software Solutions of India offers one of the Largest conglomerations of Software Training, IT Support, Corporate Training institute in India - +919000444287 - +917386622889 - Visakhapatnam,Hyderabad Hadoop Developer Online Training, Hadoop Developer Training, Hadoop Developer, Hadoop Developer Online Training| Hadoop Developer Training| Hadoop Developer| "Courses at 21st Century Software Solutions
    Talend Online Training -Hyperion Online Training - IBM Unica Online Training - Siteminder Online Training - SharePoint Online Training - Informatica Online Training - SalesForce Online Training - Many more… | Call Us +917386622889 - +919000444287 - contact@21cssindia.com
    Visit: http://www.21cssindia.com/courses.html"

    ReplyDelete
  5. Uniqe informative article and of course True words, thanks for sharing. Today I see myself proud to be a hadoop professional with strong dedication and will power by blasting the obstacles. Thanks to Big Data Training Chennai

    ReplyDelete
  6. I get a lot of great information from this blog. Thank you for your sharing this informative blog.
    AWS course chennai | AWS Certification in chennai | AWS Certification chennai

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for Learners.
    VMWare course chennai | VMWare certification in chennai | VMWare certification chennai



    ReplyDelete
  9. Nice piece of article you have shared here, my dream of becoming a hadoop professional become true with the help of Hadoop Training Chennai, keep up your good work of sharing quality articles.

    ReplyDelete
  10. Thanks for sharing your informative article on Hive ODBC Driver. Your article is very descriptive and assists me to learn whole concept in detail. Hadoop Training in Chennai

    ReplyDelete
  11. Welcome to Wiztech Automation - Embedded System Training in Chennai. We have knowledgeable Team for Embedded Courses handling and we also are after Job Placements offer provide once your Successful Completion of Course. We are Providing on Microcontrollers such as 8051, PIC, AVR, ARM7, ARM9, ARM11 and RTOS. Free Accommodation, Individual Focus, Best Lab facilities, 100% Practical Training and Job opportunities.

    Embedded System Training in chennai
    Embedded System Training Institute in chennai
    Embedded Training in chennai
    Embedded Course in chennai
    Best Embedded System Training in chennai
    Best Embedded System Training Institute in chennai
    Best Embedded System Training Institutes in chennai
    Embedded Training Institute in chennai
    Embedded System Course in chennai
    Best Embedded System Training in chennai

    ReplyDelete
  12. It seems there is no difference between the subject mentioned at this blog and hadoop online training center. Thanks for presenting the information in an excellent way.

    ReplyDelete
  13. Using big data analytics may give the companies many fruitful results, the findings can be implemented in their business decisions so as to minimize their risk and to cut the costs.
    hadoop training in chennai|big data training|big data training in chennai

    ReplyDelete
  14. Cloud computing is the next big thing, through cloud the users have the liberty to use a shared network. The companies can focus on core business parts rather than investing heavily on infrastucture.
    cloud computing training in chennai|cloud computing courses in chennai|cloud computing training

    ReplyDelete
  15. Your blog is much effective and thanks for sharing information.. Here we are providing training & materilas and dumps for those who are preparing for cca 175 spark and hadoop developer certification exam and in the related fields of cloudera hadoop developer certification exam.
    cca 175 certification
    cloudera hadoop developer certification
    cca 175 spark and hadoop developer certification
    hadoop developer certification

    ReplyDelete
  16. Hadoop Online and classroom training by real-time experts for more details visit:http://www.a1trainings.com


    Hadoop training in hyderabad



    ReplyDelete
  17. I think this is an great blogs. Such a very informative and creative contents. These concept is good for these knowledge.I like it and help me to development very well.Thank you for this brief explanations.
    Dot net Training in Chennai

    ReplyDelete
  18. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  19. Thank's For Sharing The Article....
    With regarsd@Selenium Grid

    ReplyDelete
  20. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    Best Java Training Institute Chennai

    Java Training Institute Bangalore

    ReplyDelete
  21. Ciitnoida provides Core and java training institute in

    noida
    . We have a team of experienced Java professionals who help our students learn Java with the help of Live Base Projects. The object-

    oriented, java training in noida , class-based build

    of Java has made it one of most popular programming languages and the demand of professionals with certification in Advance Java training is at an

    all-time high not just in India but foreign countries too.

    By helping our students understand the fundamentals and Advance concepts of Java, we prepare them for a successful programming career. With over 13

    years of sound experience, we have successfully trained hundreds of students in Noida and have been able to turn ourselves into an institute for best

    Java training in Noida.

    java training institute in noida
    java training in noida
    best java training institute in noida
    java coaching in noida
    java institute in noida

    ReplyDelete
  22. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.


    selenium training in bangalore

    ReplyDelete
  23. Sap Training Institute in Noida

    CIIT Noida provides Best SAP Training in Noida based on current industry standards that helps attendees to secure placements in their dream jobs at MNCs. CIIT Provides Best ERP SAP Training in Noida. CIIT is one of the most credible ERP SAP training institutes in Noida offering hands on practical knowledge and full job assistance with basic as well as advanced level ERP SAP training courses. At CIIT ERP SAP training in noida is conducted by subject specialist corporate professionals with 7+ years of experience in managing real-time ERP SAP projects. CIIT implements a blend of aERPemic learning and practical sessions to give the student optimum exposure that aids in the transformation of naïve students into thorough professionals that are easily recruited within the industry.

    At CIIT’s well-equipped ERP SAP training center in Noida aspirants learn the skills for ERP SAP Basis, ERP SAP ABAP, ERP SAP APO, ERP SAP Business Intelligence (BI), ERP SAP FICO, ERP SAP HANA, ERP SAP Production Planning, ERP SAP Supply Chain Management, ERP SAP Supplier Relationship Management, ERP SAP Training on real time projects along with ERP SAP placement training. ERP SAP Training in Noida has been designed as per latest industry trends and keeping in mind the advanced ERP SAP course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies and achieve their career goals.

    ReplyDelete
  24. BCA Colleges in Noida

    CIIT Noida provides Sofracle Specialized B Tech colleges in Noida based on current industry standards that helps students to secure placements in their dream jobs at MNCs. CIIT provides Best B.Tech Training in Noida. It is one of the most trusted B.Tech course training institutes in Noida offering hands on practical knowledge and complete job assistance with basic as well as advanced B.Tech classes. CIITN is the best B.Tech college in Noida, greater noida, ghaziabad, delhi, gurgaon regoin .

    At CIIT’s well-equipped Sofracle Specialized M Tech colleges in Noida aspirants learn the skills for designing, analysis, manufacturing, research, sales, management, consulting and many more. At CIIT B.Tech student will do practical on real time projects along with the job placement and training. CIIT Sofracle Specialized M.Tech Classes in Noida has been designed as per latest IT industry trends and keeping in mind the advanced B.Tech course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies (MNCs) and achieve their career goals.

    MCA colleges in Noida we have high tech infrastructure and lab facilities and the options of choosing multiple job oriented courses after 12th at Noida Location. CIIT in Noida prepares thousands of engineers at reasonable B.Tech course fees keeping in mind training and B.Tech course duration and subjects requirement of each attendee.

    Engineering College in Noida"

    ReplyDelete
  25. This concept is a good way to enhance the knowledge.thanks for sharing. please keep it up salesforce Online Training Bangalore

    ReplyDelete


  26. Hi Your Blog is very nice!!

    Get All Top Interview Questions and answers PHP, Magento, laravel,Java, Dot Net, Database, Sql, Mysql, Oracle, Angularjs, Vue Js, Express js, React Js,
    Hadoop, Apache spark, Apache Scala, Tensorflow.

    Mysql Interview Questions for Experienced
    php interview questions for freshers
    php interview questions for experienced
    python interview questions for freshers
    tally interview questions and answers






    ReplyDelete
  27. Nice to read this article... Thanks for sharing

    DevOps Online Training

    ReplyDelete