Hadoop Assignment

You are given 2 CSV data sets: (a) A course dataset containing details of courses offered (b) A job description dataset containing a list of job descriptions (Note: Each field of a job description record is demarcated by " ") You have to design and implement a distributed recommendation system using the data sets, which will recommend the best courses for up-skilling based on a given job description. You can use the data set to train the system and pick some job descriptions not in the training set to test. It is left up to you how you pick necessary features and build the training that creates matching courses for job profiles. These are the suggested steps you should follow : Step 1: Setup a Hadoop cluster where the data sets should be stored on the set of Hadoop data nodes. Step 2: Implement a content based recommendation system using MapReduce, i.e. given a job description you should be able to suggest a set of applicable courses. Step 3: Execute the training step of your MapReduce program using the data set stored in the cluster. You can use a subset of the data depending on the system capacity of your Hadoop cluster. You have to use an appropriate subset of features in the data set for effective training. Step 4: Test your recommendation system using a set of requests that execute in a distributed fashion on the cluster. You can pick a set of 3-5 job descriptions in the data set to show how they are executed in parallel to provide corresponding course recommendations.