Data science python - implement feature selection from large CSV

Implement feature selection from a very large dimensionality dataset: You need to implement one function in python. The input to the function is: 1. string - Path to a csv file 2. integer - n_dimensions - Desired number of output dimensions The function needs to select the best n_dimensions by fitting RandomForest **iteratively** classifier and selecting the features based on feature_importances_ . See: [login to view URL] IMPORTANT: 1. The size of the csv is very large: might be >5gb and the number of dimensions >1M. Therefore, your main task in this project is to find the best chunk size to read the data from the CSV and train the RandomForest classifier, so the function will work in any circumstances, even with very large input files. In other words, if the number of output dimensions is 10K and the number of input dimensions is 1M, the optimal way might be to fit 20 RandomForests, each with 50K, select 10K from each. Then - merge the 200K dimensions selected in the first iteration, and fit another 4 RandomForests, each with 50K. In the last iteration, fit one RandomForest with 40K and finally select the last 10K. 2. Regarding how to select how many iterations and how many random forests to fit: These could be extra parameters to the function. I also expect you to research and recommend best parameters based on execution time and memory usage. The function needs to run well on the client side which is normally a windows 10 pc with 4-8Gb RAM and 2-4 cores. Reading the input into numpy arrays should be done smartly, without loading the entire csv to memory. (maybe numpy has this required functionality built-in) 3. The function should support both parallel and serial modes. In parallel mode, the function should utilize N cores of the PC, by fitting several RandomForests on parallel, each on a separate core. In serial mode the function should fit the RandomForests one by one. 4. The function should be able to execute on python 3.5 environment on both windows. You must test it on windows! 5. The function should output to the console its progress 6. I will send example csv to coders with good experience