MapReduce : Simplified Data Processing on Large Cluster

  • Muthu Dayalan Senior Software Developer, ANNA University, India

Abstract

MapReduce is a data processing approach, where a single machine acts as a master, assigning map/reduce tasks to all the other machines attached in the cluster. Technically, it could be considered as a programming model, which is applied in generating, implementation and generating large data sets. The key concept behind MapReduce is that the programmer is required to state the current problem in two basic functions, map and reduce. The scalability is handles within the system, rather than being handled by the concerned programmer. By applying various restrictions on the applied programming style, MapReduce performs several moderated functions such fault tolerance, locality optimization, load balancing as well as massive parallelization. Intermediate k/v pairs are generated by the Map, and then fed o the reduce workers by the use of the incorporated file system. The data received by the reduce workers is then merged using the same key, to produce multiple output file to the concerned user (Dean & Ghemawat, 2008). Additionally, the programmer is only required to master and write the codes regarding the easy to understand functionality.

Downloads

Download data is not yet available.

References

[1] Dean, J., & Ghemawat, S. (2008). MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1), 107-113.
[2] Dean, J., & Ghemawat, S. (2010). MapReduce: a flexible data processing tool. Communications of the ACM, 53(1), 72-77.
[3] Kolpin, G. (2006). MapReduce: Simplified Data Processing on Large Clusters.
[4] McKenna, A., Hanna, M., Banks, E., Sivachenko, A., Cibulskis, K., Kernytsky, A., ... &
[5] DePristo, M. A. (2010). The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. Genome research, 20(9), 1297-1303.
[6] Yang, H. C., Dasdan, A., Hsiao, R. L., & Parker, D. S. (2007, June). Map-reduce-merge:
[7] simplified relational data processing on large clusters. In Proceedings of the 2007 ACM SIGMOD international conference on Management of data(pp. 1029-1040). ACM.
[8] Bin Saadon, A.,G., & Mokhtar, H. M. O. (2017). iiHadoop: An asynchronous distributed framework for incremental iterative computations. Journal of Big Data, 4(1), 1-30. doi:http://dx.doi.org/10.1186/s40537-017-0086-3
[9] Boja, C., Pocovnicu, A., & Batagan, L. (2012). Distributed parallel architecture for "big data". Informatica Economica, 16(2), 116-127. Retrieved from https://search.proquest.com/docview/1030278615?accountid=45049
[10] Chang, B., Tsai, H., Tsai, Y., Kuo, C., & Chen, C. (2016). Integration and optimization of multiple big data processing platforms. Engineering Computations, 33(6), 1680-1704. Retrieved from https://search.proquest.com/docview/1802745541?accountid=45049
[11] Chen, L., Liu, Y., Gallagher, M., Pailthorpe, B., Sadiq, S., Shen, H. T., & Li, X. (2012). Introducing cloud computing topics in curricula. Journal of Information Systems Education, 23(3), 315-324. Retrieved from https://search.proquest.com/docview/1315921601?accountid=45049
[12] Ding, S., Li, G., Li, Y., Li, X., Zhai, Q., Champion, A. C., . . . Zheng, Y. F. (2017). SurvSurf: Human retrieval on large surveillance video data. Multimedia Tools and Applications, 76(5), 6521-6549. doi:http://dx.doi.org/10.1007/s11042-016-3307-4
[13] Hare, J. S., Samangooei, S., & Lewis, P. H. (2014). Practical scalable image analysis and indexing using hadoop. Multimedia Tools and Applications, 71(3), 1215-1248. doi:http://dx.doi.org/10.1007/s11042-012-1256-0
[14] Islam, A. K., M, T., Jeong, B., Bari, A. T., M, . . . Jeon, S. (2015). MapReduce based parallel gene selection method. Applied Intelligence, 42(2), 147-156. doi:http://dx.doi.org/10.1007/s10489-014-0561-x
[15] Lamari, Y., & Said, C. S. (2017). Clustering categorical data based on the relational analysis approach and MapReduce. Journal of Big Data, 4(1), 1-16. doi:http://dx.doi.org/10.1186/s40537-017-0090-7
[16] Landset, S., Khoshgoftaar, T. M., Richter, A. N., & Hasanin, T. (2015). A survey of open source tools for machine learning with big data in the hadoop ecosystem. Journal of Big Data, 2(1), 1-36. doi:http://dx.doi.org/10.1186/s40537-015-0032-1
[17] Liu, X., Wang, X., Matwin, S., & Japkowicz, N. (2015). Meta-MapReduce for scalable data mining. Journal of Big Data, 2(1), 1-21. doi:http://dx.doi.org/10.1186/s40537-015-0021-4
[18] Mishra, P., & Somani, A. K. (2017). Host managed contention avoidance storage solutions for big data. Journal of Big Data, 4(1), 1-42. doi:http://dx.doi.org/10.1186/s40537-017-0080-9
[19] Mohamed, H., & Marchand-maillet, S. (2014). Distributed media indexing based on MPI and MapReduce. Multimedia Tools and Applications, 69(2), 513-537. doi:http://dx.doi.org/10.1007/s11042-012-1283-x
[20] Nagwani, N. K. (2015). Summarizing large text collection using topic modeling and clustering based on MapReduce framework. Journal of Big Data, 2(1), 1-18. doi:http://dx.doi.org/10.1186/s40537-015-0020-5
[21] Najafabadi, M. M., Khoshgoftaar, T. M., Villanustre, F., & Holt, J. (2017). Large-scale distributed L-BFGS. Journal of Big Data, 4(1), 1-17. doi:http://dx.doi.org/10.1186/s40537-017-0084-5
[22] Rathee, S., & Kashyap, A. (2018). Adaptive-miner: An efficient distributed association rule mining algorithm on spark. Journal of Big Data, 5(1), 1-17. doi:http://dx.doi.org/10.1186/s40537-018-0112-0
[23] Sharma, S., & Toshniwal, D. (2017). Scalable two-phase co-occurring sensitive pattern hiding using MapReduce. Journal of Big Data, 4(1), 1-18. doi:http://dx.doi.org/10.1186/s40537-017-0064-9
[24] Singh, D., & Reddy, C. K. (2014). A survey on platforms for big data analytics.Journal of Big Data, 2(1), 1-20. doi:http://dx.doi.org/10.1186/s40537-014-0008-6
[25] Suthakar, U., Magnoni, L., David, R. S., Khan, A., & Andreeva, J. (2016). An efficient strategy for the collection and storage of large volumes of data for computation. Journal of Big Data, 3(1), 1-17. doi:http://dx.doi.org/10.1186/s40537-016-0056-1
[26] Trifu, M. R., & Ivan, M. (2016). Big data components for business process optimization. Informatica Economica, 20(1), 72-78. doi:http://dx.doi.org/10.12948/issn14531305/20.1.2016.07
[27] Wang, H., Zhu, F., Xiao, B., Wang, L., & Jiang, Y. (2015). GPU-based MapReduce for large-scale near-duplicate video retrieval. Multimedia Tools and Applications, 74(23), 10515-10534. doi:http://dx.doi.org/10.1007/s11042-014-2185-x
Published
2018-06-04
How to Cite
DAYALAN, Muthu. MapReduce : Simplified Data Processing on Large Cluster. International Journal of Research and Engineering, [S.l.], v. 5, n. 5, p. 399-403, june 2018. ISSN 2348-7860. Available at: <https://digital.ijre.org/index.php/int_j_res_eng/article/view/339>. Date accessed: 15 sep. 2019. doi: https://doi.org/10.21276/ijre.2018.5.5.4.