Wednesday, July 10, 2013

Speed up hadoop development with progressive testing

Debugging Hadoop jobs can be a huge pain.  The cycle time is slow, and error messages are often uninformative --- especially if you're using Hadoop streaming, or working on EMR.

I once found myself trying to debug a job that took a full six hours to fail.  It took more than a week -- a whole week! -- to find and fix the problem.  Of course, I was doing other things at the same time, but the need to constantly check up on the status of the job was a huge drain on my energy and productivity.  It was a Very Bad Week.



Painful experiences like this have taught me to follow a test-driven approach to hadoop development.  Whenever I'm working on a new hadoop-based data pipe, my goal is to isolate six distinct kinds of problems that arise in hadoop development.

  1. Explore the data: The pipe must accept data from a given format, which might not be fully understood at the outset.
  2. Test basic logic: The pipe must execute the intended data transformation for "normal" data. 
  3. Test edge cases: The pipe must deal gracefully with edge cases, missing or misformatted fields, rare divide-by-zeroes, etc. 
  4. Test deployment parameters: The pipe must be deployable on hadoop, with all the right filenames, code dependencies, and permissions.
  5. Test cluster performance: For big enough jobs, the pipe must run efficiently.  If not, we need to tune or scale up the cluster.
  6. Test scheduling parameters: Once pipes are built, routine jobs must be scheduled and executed.

Each of these steps requires different test data and different methods for trapping and diagnosing errors.  Therefore, the goal is to make sure to (1) tackle problems one at a time, and (2) solve each kind of problem in the environment with the fastest cycle time.

Steps 1 through 3 should be solved locally, using progressively larger data sets.  Steps 4 and 5 must be run remotely, again using progressively larger data sets.

Step 6 depends on your scheduling system and has a very slow cycle time (i.e. you must wait a day to test whether your daily jobs run on the proper schedule.).  However, it's independent of hadoop, so you can build, test, and deploy it separately.  (There may be some crossover with #4, but you can test this with small data sets.)

Going through six different rounds of testing may seem like overkill, but in my experience it's absolutely worth it.  Very likely, you'll encounter at least one new bug/mistake/unanticipated case at each stage.  Progressive testing ensures that each bug is dealt with as quickly as possible, and prevents them from ganging up on you.

Other suggestions:
  • Definitely use an abstraction layer that allows you to seamlessly deploy local code to your staging and production clusters.  Cascalog and mrJob are good examples.  Otherwise, you'll find yourself solving steps 2 and 3 all over again in deployment.
  • Config files and object-oriented code can reduce a lot of headaches in step 4.  Most of your deployment hooks can be written once and saved in a config file.  If you have strong naming conventions, then most of your filenames can be constructed (and tested) programmatically.  It's amazing how many hours you can waste debugging a simple typo in hadoop.  Good OOP will spare you many of these headaches.
  • Part of the beauty of Hive and HBase is that they abstract away most of the potential pitfalls on the deployment side, especially in step 4.  By the same token, tools like Azkaban and Oozie can take a lot of the pain out of step 6.  (Be careful, though -- each of these scheduling tools has its limitations.)

28 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. Uniqe informative article and of course True words, thanks for sharing. Today I see myself proud to be a hadoop professional with strong dedication and will power by blasting the obstacles. Thanks to Big Data Training Chennai

    ReplyDelete
  3. I get a lot of great information from this blog. Thank you for your sharing this informative blog.
    AWS course chennai | AWS Certification in chennai | AWS Certification chennai

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for Learners.
    VMWare course chennai | VMWare certification in chennai | VMWare certification chennai



    ReplyDelete
  6. Nice piece of article you have shared here, my dream of becoming a hadoop professional become true with the help of Hadoop Training Chennai, keep up your good work of sharing quality articles.

    ReplyDelete
  7. Thanks for sharing your informative article on Hive ODBC Driver. Your article is very descriptive and assists me to learn whole concept in detail. Hadoop Training in Chennai

    ReplyDelete
  8. It seems there is no difference between the subject mentioned at this blog and hadoop online training center. Thanks for presenting the information in an excellent way.

    ReplyDelete
  9. Using big data analytics may give the companies many fruitful results, the findings can be implemented in their business decisions so as to minimize their risk and to cut the costs.
    hadoop training in chennai|big data training|big data training in chennai

    ReplyDelete
  10. Cloud computing is the next big thing, through cloud the users have the liberty to use a shared network. The companies can focus on core business parts rather than investing heavily on infrastucture.
    cloud computing training in chennai|cloud computing courses in chennai|cloud computing training

    ReplyDelete
  11. I think this is an great blogs. Such a very informative and creative contents. These concept is good for these knowledge.I like it and help me to development very well.Thank you for this brief explanations.
    Dot net Training in Chennai

    ReplyDelete
  12. Thank's For Sharing The Article....
    With regarsd@Selenium Grid

    ReplyDelete
  13. Ciitnoida provides Core and java training institute in

    noida
    . We have a team of experienced Java professionals who help our students learn Java with the help of Live Base Projects. The object-

    oriented, java training in noida , class-based build

    of Java has made it one of most popular programming languages and the demand of professionals with certification in Advance Java training is at an

    all-time high not just in India but foreign countries too.

    By helping our students understand the fundamentals and Advance concepts of Java, we prepare them for a successful programming career. With over 13

    years of sound experience, we have successfully trained hundreds of students in Noida and have been able to turn ourselves into an institute for best

    Java training in Noida.

    java training institute in noida
    java training in noida
    best java training institute in noida
    java coaching in noida
    java institute in noida

    ReplyDelete
  14. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.


    selenium training in bangalore

    ReplyDelete
  15. BCA Colleges in Noida

    CIIT Noida provides Sofracle Specialized B Tech colleges in Noida based on current industry standards that helps students to secure placements in their dream jobs at MNCs. CIIT provides Best B.Tech Training in Noida. It is one of the most trusted B.Tech course training institutes in Noida offering hands on practical knowledge and complete job assistance with basic as well as advanced B.Tech classes. CIITN is the best B.Tech college in Noida, greater noida, ghaziabad, delhi, gurgaon regoin .

    At CIIT’s well-equipped Sofracle Specialized M Tech colleges in Noida aspirants learn the skills for designing, analysis, manufacturing, research, sales, management, consulting and many more. At CIIT B.Tech student will do practical on real time projects along with the job placement and training. CIIT Sofracle Specialized M.Tech Classes in Noida has been designed as per latest IT industry trends and keeping in mind the advanced B.Tech course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies (MNCs) and achieve their career goals.

    MCA colleges in Noida we have high tech infrastructure and lab facilities and the options of choosing multiple job oriented courses after 12th at Noida Location. CIIT in Noida prepares thousands of engineers at reasonable B.Tech course fees keeping in mind training and B.Tech course duration and subjects requirement of each attendee.

    Engineering College in Noida"

    ReplyDelete


  16. Hi Your Blog is very nice!!

    Get All Top Interview Questions and answers PHP, Magento, laravel,Java, Dot Net, Database, Sql, Mysql, Oracle, Angularjs, Vue Js, Express js, React Js,
    Hadoop, Apache spark, Apache Scala, Tensorflow.

    Mysql Interview Questions for Experienced
    php interview questions for freshers
    php interview questions for experienced
    python interview questions for freshers
    tally interview questions and answers






    ReplyDelete
  17. Really very happy to say, your post is very interesting to read. I never stop myself to say something about it. You’re doing a great job. Keep it up…

    Become an Expert In SAP BASIS Training! The most trusted and trending Programming Language. Learn from experienced Trainers and get the knowledge to crack a coding interview, @Softgen Infotech Located in BTM Layout.

    ReplyDelete
  18. Interested in the DevOps Training in Chennai with Placement opportunities? Infycle Technologies will help you for getting an excellent DevOps training in practical teaching methods along with 100% placement guidance. To get your first DevOps job with the highest salary package, get a free demo from us by dialing 7504633633 or 7502633633!

    ReplyDelete
  19. Casino and Sports Betting - DRMCD
    This 밀양 출장안마 page is not intended 과천 출장안마 for commercial use or for financial purposes. 김해 출장안마 It may harm your computer, work, or access personal data. 남원 출장샵 Casino and Sports Betting. 진주 출장마사지

    ReplyDelete