About this course
Apache Hadoop is one of the most popular tools for big data processing. It has been successfully deployed in production by many companies for several years. Though Hadoop is considered a reliable, scalable, and cost-effective solution, it is constantly being improved by a large community of developers. As a result, the 2.0 version offers several revolutionary features, including Yet Another Resource Negotiator (YARN), HDFS Federation, and high availability, which make the Hadoop cluster much more efficient, powerful, and reliable.
The most serious limitations of classical MapReduce are primarily related to scalability, resource utilization, and the support of workloads different from MapReduce. In the MapReduce framework, the job execution is controlled by two types of processes: a single master process called JobTracker and a number of subordinate processes called TaskTrackers.
Apache Hadoop 2.0 includes YARN, which separates the resource management and processing components. The YARN-based architecture is not constrained to MapReduce. In YARN, MapReduce is simply degraded to a role of a distributed application (but still a very popular and useful one) and is now called MRv2. MRv2 is simply the re-implementation of the classic MapReduce engine, now called MRv1, which runs on top of YARN.
The course reviews MapReduce1 and provides insight into the design and implementation of YARN: ResourceManager instead of a cluster manager, ApplicationMaster instead of a dedicated and short-lived JobTracker, NodeManager instead of TaskTracker, a distributed application instead of a MapReduce job.
Glen R.J. Mules
Glen R.J. Mules is a Senior Instructor and Principal Consultant with IBM Information Management World-Wide Education and works from New Rochelle, NY. Glen joined IBM in 2001 as a result of IBM's acquisition of Informix Software. He has worked at IBM, and previously at Informix Software, as an instructor, a course developer, and in the enablement of instructors worldwide. He teaches courses in BigData (BigInsights & Streams), Optim, Guardium, and DB2, & Informix databases. He has a BSc in Mathematics from the University of Adelaide, South Australia; an MSc in Computer Science from the University of Birmingham, England; and has just completed a PhD in Education (Educational Technology) at Walden University. His early work life was as a high school teacher in Australia. In the 1970s he designed, programmed, and managed banking systems in Manhattan and Boston. In the 1980s he was a VP in Electronic Payments for Bank of America in San Francisco and New York. In the early 1990s he was an EVP in Marketing for a software development company and chaired the ANSI X12C Standards Committee on Data Security for Electronic Data Interchange (EDI).
Joe Byers is a Senior Technical Curriculum Developer with IBM World-Wide Education and develops training on various media formats for Business Intelligence, Predictive Analytics, and Information Management. Joe came to IBM in 2007. Prior to that he was a Technical Manager for Oracle Corporation, where he spent 10 years engineering and architecting data warehouses. Joe performed his undergraduate and graduate studies in Computer Science at Indiana University, after which, he became a database programmer for Sears Craftsman Tools in Blue Ash, Ohio. In the 1990s, prior to joining Oracle, Joe was the Data Administrator at The Andrew Jergens Company in Cincinnati, Ohio, responsible for the entire company's data. While at Andrew Jergens, Joe became an Oracle Master and earned his Oracle Certified DBA (OCP). Joe regularly leads BI sessions at IBM's annual Insight conference in Las Vegas. Joe's other professional certifications include five different certifications in IBM Cognos products, two certifications in IBM TM1, as well as certifications and badges in IBM DB2, SPARK, Hadoop, and IBM Big Data.