tag:blogger.com,1999:blog-4794900363555082282024-02-07T03:58:32.060-08:00Suresh KaukuntlyUnknownnoreply@blogger.comBlogger4125tag:blogger.com,1999:blog-479490036355508228.post-17676601545702149982020-05-20T12:11:00.004-07:002020-05-20T21:04:25.909-07:00Purusharthas: The human desires<div>Sanatana Dharma and numbers have a great relation. Below are miniscule few of them,</div><div><br /></div><div>0. Sunya state of meditator</div><div>1. Brahman</div><div>2. Shiva and Shakthi</div><div>3. Trimurti</div><div>4. Purushartas (The topic of discussion here)</div><div>5. Pancha prana. Pancha kosha</div><div>6. Arishadvargas</div><div>7. Saptha rishis</div><div>8. Astalakshmi</div><div>9. Navagrahas</div><div>10. Dashavataras</div><div>11. 11 Chakras ( As taught by my guru Master Choa kok sui)</div><div>12. 12 Chakra ( Seat of Incarnated Soul/Jivatma, as As taught by my guru Master Choa kok sui)</div><div><br /></div><div>I always get marvelled by Sanatana Dharma, Its details on topic of the purpose of our existence is astounding. Today we will learn about purusharthas. One of the term least understood. Earlier reference to this word can be found </div><div><br /></div><div>1. Suryopanishad </div><div><br /></div><div>aum atha sūryātharvāṅgirasam vyā̎khyāsyā̱maḥ | brahmā ṟṣih | gāya̍trī chha̱ndaḥ | ādi̍tyo de̱vatā | haṁsa̍ḥ so̱’ham agni nārāyana yu̍ktam bī̱jam | hrille̍khā śa̱ktiḥ | viyadādi sarga saṁyu̍ktaṁ kī̱lakam | catur-vidha puruṣārtha siddhyarthe jape vi̍niyo̱gaḥ ||</div><div><br /></div><div>2. Ganapty Atharvashrisha Upanishad</div><div><br /></div><div>सायमधीयानो दिवसकृतं पापं नाशयति । प्रातरधीयानो रात्रिकृतं पापं नाशयति । सायं प्रातः प्रयुञ्जानो पापोऽपापो भवति । सर्वत्राधीयानोऽपविघ्नो भवति । धर्मार्थकाममोक्षं च विन्दति ॥१३॥</div><div><br /></div><div>Purushartha is compound word formed by Purusha + Artha. Purusha in sanatana dharms stands for the soul. Artha has many meanings,</div><div><br /></div><div>1. Goal</div><div>2. That which is acquired</div><div>3. Prosperity</div><div><br /></div><div>Traditionally Purushartha means human goals. Its very difficult to understand purushartha as goals. For discussion we will go with much easier explanation which is "Desires". Purushartha stands for 4 kinds of desires which drive human evolution, without which the world is stagnant.</div><div><br /></div><div>1. Dharma: The desire to be right. Dharma is the one which makes Scientists,Mathematicians,Physicists,Chemists to experiment and verify their understanding. Dharma is the one which makes people standup for their beliefs. Dharma is the one which make kid argue 1 + 1 = 2 and not 1 + 1 = 10, But 10th Standard student will argue that both are true in its own sense, one is base 2, one in base 10. Dharma is the ultimate desire to be right. For humans, we have limitations of the understanding of universe, But we try to be right within our own little understanding. As we expand our knowledge and understanding, our Actions also change. This is because of desire to be right, Which we call as Dharma.</div><div><br /></div><div>Satya yuga has many incidents where divinity trumps the bad. All of these stories are centered around Ego. Take for example of Hayagriva avatar or Narasimha avatar, The villains in both the stories were destroyed because of their Ego. They thought they are the one who is right. They thought themselves as ultimate knower of the truth.</div><div><br /></div><div>All these stories hints us one common thing. Humans in Satya yuga had capabilities of only Dharma, Or could desire to be right only. They were incapable of any other desires like Artha, Kaama or Moksha. All their actions were because of desire to be right, Whether its to acquire wealth, Procreation, or any other actions. All were cause of desire to be right. They thought its right thing to do to acquire wealth, its right thing to procreate.</div><div><br /></div><div>Humans are evolved enough to fulfill dharma, Desire to be right.</div><div><br /></div><div>2. Artha: Desire to possess. Artha is the one which makes humans put effort to acquire. Be it Wealth. Be it name or fame. Be it beautiful house, car, kingdom, desire for any form of possession is because of Artha. </div><div><br /></div><div>Humans in Treta yuga had capabilities of both kinds of desires, Dharma and Artha. All of tretha yuga incidence are mostly centered around acquiring. But both Dharma and Artha are responsible for happenings in Treta yuga. Bali chakravarthi wanted to acquire all lokas and liberated by Vaamana avatar, Vali wanted Sugrivas wife, Sita wanted golden deer, Raavana wanted Sita. Kaikeyi wanted kingdom to her son. All were cause of desire to acquire. They were primarly motivated by Artha. </div><div><br /></div><div>Humans are evolved enough to fulfill both dharma and artha</div><div><br /></div><div>3. Kaama: Desire to experience. Kaama is the one which makes humans want to experience. Humans jump from flight to experience the thrill, Go for scuba diving, Have Sex, Get angry, Show empathy, Eat delicious food, Drink what they enjoy, All because of Kaama, Desire to experience.</div><div><br /></div><div>Humans in Dwapara yuga additionaly had capabilities of kaama, i.e desire to experience. The mahabharata was result of kaama, Duryodhanas hate towards his brothers. He enjoyed hating. This is possible only because humans now could experience and desire for it. However all three, Dharma, Artha and Kaama are responsible for happenings in Dwaapara yuga. Kaama being most prominent.</div><div><br /></div><div>Humans are evolved enough to fulfill Dharma, Artha and Kaama by now</div><div><br /></div><div>4. Moksha: Desire to unite with the one. Moksha is the one which makes humans want to unite with the Brahman. We worship, We meditate, We want to be spiritual, We want to be religious, We want to die in the name of god, We want to kill in the name of god, We want to convert and destroy society in the name of god. All these are fuelled due to desire to unite with God. However humans are now balancing all four purusharthas. All four kinds of desires. </div><div><br /></div><div>Humans are evolved enough to fulfill Dharma, Artha, Kaama and Moksha by now.</div><div><br /></div><div>I hope you enjoyed reading this and have fair understanding of purusharthas, You can now analyse your actions and see what motivates them, Dharma, artha, kaama or moksha</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-479490036355508228.post-68887197680361740712020-04-29T19:16:00.000-07:002020-04-29T19:16:54.876-07:00Karma: A Bhagvatgita view<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
There are very good answers to this question here. However i will try to present my thoughts little differently. Bhagavat Gita is most practical book related to spirituality. There is no philosophical views expressed in Gita, Every sloka explains the science behind the working of the universe. Krishna never propagates any philosophy, he merely reveals the inner working, Its upto the reader to figure out how to come out of the cycle of birth and death by figuring out a own way.</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Now coming to the topic in discussion, many slokas talks about karma yoga, most famous would be the below from chapter 2 verse 47</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Karmanyevaadhikaaraste maa phaleshu kadaachana;</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Maa karmaphalahetur bhoor maa te sango’stwakarmani.</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Literal sense is very philosophical, and bhagvat gita is anything but philosophy. Its a science. So to understand and appreciate this sloka, we need to have little knowledge about karma and its working.</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Every “Action” produces 2 types of “Phala”</div>
<ol class="q-box" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; list-style-image: initial; list-style-position: initial; margin: 0px 2em 1em 0px; padding: 0px;">
<li style="margin: 0px 0px 0px 2em; padding: 0px 0px 0.7em; position: relative;">Drsta Phala: Which can be perceived, Like Salary you get for your work, appreciations when you help someone etc.</li>
<li style="margin: 0px 0px 0px 2em; padding: 0px; position: relative;">Adrsta Phala: This gets into your debit credit account, also called Karma generally</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://qphs.fs.quoracdn.net/main-qimg-99f14fa4b34fed43f5a1e590b4b8b336.webp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<div class="CssComponent-sc-1oskqb9-0 QTextImage___StyledCssComponent-sc-1yi3aau-0 ilDNcJ" style="background-color: white; color: #282829; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px;">
<div class="q-box unzoomed" style="box-sizing: border-box; cursor: -webkit-zoom-in; direction: ltr; filter: blur(0px); margin-bottom: 1em;">
</div>
</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Moreover Drsta phala is result of your current action and also previous adrsta phala as shown below<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHPwzSy3Ss075Jj0OSqRHWsrbzfZzp-b85zgc0P-kPjn25MHE59M62QutQRfd8Q6HIvu2lKOJsqI-9S_Wd9EKeBWooi3z9ftrXaZVslrOEPEw6hB6opgfNlIA_LQrPZr3DayqPXpcw2wA/s1600/karma.webp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="237" data-original-width="379" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHPwzSy3Ss075Jj0OSqRHWsrbzfZzp-b85zgc0P-kPjn25MHE59M62QutQRfd8Q6HIvu2lKOJsqI-9S_Wd9EKeBWooi3z9ftrXaZVslrOEPEw6hB6opgfNlIA_LQrPZr3DayqPXpcw2wA/s320/karma.webp" width="320" /></a></div>
<br />
<br />
<br />
Mathematically</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Drsta Phala = Current Action + Adrsta Phala (Karma)</div>
<div class="q-text qu-display--block" style="background-color: white; box-sizing: border-box; color: #282829; direction: ltr; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 15px; margin-bottom: 1em; overflow-wrap: break-word; padding: 0px;">
Technically speaking, You have control of only Current Action, do not have any control on past action(Karma), So you have no control(Rights to decide) on the result, The best you can do is give your best effort on current action and leave the unknown to the god. Thats the reason this sloka talks about Adhikara(Authority) and not about dharma(Duty). I feel this is most misunderstood concept of karma yoga and specially this sloka. Karma yoga is karmic science, Krishna explains how karma works, When it binds, When it doesn’t, etc. He never tells this is the right way to be, Its just science what he explains, readers can apply it as they wish.</div>
</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-479490036355508228.post-16860032840556369622015-01-23T01:32:00.000-08:002015-01-23T02:03:32.498-08:00Data Systems<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Generally all Data Systems have below components:</b><br />
<br />
1. Data Collection:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Enables feeding data into the system. Copy log files, sync with external database etc<br />
<br />
2. Data Storage<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Stores data. Can be disk based, or Database too.<br />
<br />
3. Processing<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Processes the large volume of data. Data can be stored on disk or database.<br />
<br />
4. Querying<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Fetch the data from Data Storage with appropriate filters.<br />
<br />
5. Reporting<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Display/Visualize/Generate reports Information derived from the Data.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCShpw65B86XzBxAoVbkFpItrkvJyJeoRtIzCRZaDAy1lNtUikyMRZf3IJV0WtEr8X80zjegpn1fK5tl4M-Fwzgg3Yl0o5qMYhxF1E8_9qyV51M33PrK0jx29fci_CGuYre47dmSjO870/s1600/DataSystems.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCShpw65B86XzBxAoVbkFpItrkvJyJeoRtIzCRZaDAy1lNtUikyMRZf3IJV0WtEr8X80zjegpn1fK5tl4M-Fwzgg3Yl0o5qMYhxF1E8_9qyV51M33PrK0jx29fci_CGuYre47dmSjO870/s1600/DataSystems.png" height="640" width="544" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Data Systems components</td></tr>
</tbody></table>
<br />
What differentiates <b>BigData </b>systems with <b>Traditional </b>Data Systems is <i style="font-weight: bold;">Scalability. </i>Traditional systems satisfied the requirement of the past era where a single installation of database could meet the requirement of the whole organization. In the age of internet, Such systems do not cater to process webscale of data.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBdQSAt_WWRoV8LsgDaVuL_ISAitq0wN_r6IJLZ2Z6vrui9_tOvI5Vwvc8YhPcGGbUQbJhEdddXdHJKykQo3onhT3wDqYWnqnGg31k_iwOhKPkvsCvNu6iZL-5UR_OTke81HEWJspvuZk/s1600/Difference.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBdQSAt_WWRoV8LsgDaVuL_ISAitq0wN_r6IJLZ2Z6vrui9_tOvI5Vwvc8YhPcGGbUQbJhEdddXdHJKykQo3onhT3wDqYWnqnGg31k_iwOhKPkvsCvNu6iZL-5UR_OTke81HEWJspvuZk/s1600/Difference.png" height="354" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Difference b/w Traditional data systems and BigData systems</td></tr>
</tbody></table>
<br />
<b>BigData systems:</b><br />
<br />
There are many BigData systems available, below is the list of few well known systems.<br />
<br />
1. Hadoop: Developed and opensourced by Yahoo!<br />
2. Cosmos: Microsoft BigData system<br />
3. Oracle Big Data Appliance: Hardware and Software combo from Oracle.<br />
<br />
<br />
Among these, Hadoop is well known opensource BigData ecosystem.<br />
<div>
<br /></div>
<div>
<br /></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-479490036355508228.post-9671298728712067562014-06-19T03:19:00.000-07:002015-01-23T06:08:34.935-08:00Introduction to BigData<div dir="ltr" style="text-align: left;" trbidi="on">
The "Big" in BigData literally signifies size of the data at hand.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Examples:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>1. Data generated due to Billions of likes by Facebook users every day<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>2. Data generated due to Millions of tweets<br />
<br />
This data will result in huge amount of back end data sizing up to terabytes, Hence the name BigData. Along with Size, The rate at which the data is generated also is high. Also the variety of data that is generated, May be unstructured log files, or structured ones. So 3 V's or properties which describe the BigData are, Volume, Velocity and Variety. Although BigData pundits define it with 3 V's, The first 'V' i.e Volume itself is enough for it to be called as BigData, Velocity and Variety are result of the source of the data and should not be a prerequisite for considering any BigData solutions.<br />
<br />
<b>Lets look at processing BigData:</b><br />
<br />
Processing BigData requires <b>"Divide and Conquer"</b> approach.<br />
<br />
For example, Suppose we have 1 K records to search for some specific word(grep), and to check 1 K line it takes around 1 second, how long would it take to grep 1 billion lines ?<br />
<br />
If we assume we will have a dedicated system, it will take 1,000,000 seconds to process 1,000,000,000 (1 Billion) records, i.e approximately around 11 days.<br />
<br />
So what? We can wait for 11 days, Whats the issue? Problem is in those 11 days, We would have generated billions of more records for grepping, this is a never ending process as more data is generated continuously.<br />
<br />
<b>Whats the solution?</b><br />
<br />
1. Build a super fast super computer, so grepping is faster than the data generation<br />
2. Setup multiple computer to process 1 Billion lines and parallelize the processing so grepping is faster than the data generation<br />
3. If our requirement is to only get the count of lines having the specified word, we can have statistical way. Grep only 100000 lines for the specific word, and then extrapolate for 1 Billion lines.<br />
<br />
Currently option 2 is lot cheaper than 1 and certain compared to option 3. Also option 3 is applicable only when we need counts or averages and not suitable in other text processing.<br />
<br />
To implement option 2, Below steps provide a <b>Golden Template</b> which is <b>Divide and Conquer</b> method for handling such processing.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>1. Setup multiple system in network to process the BigData<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>2. Divide the BigJob to smaller Job Parts<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>3. Distribute the Jobs to individual systems<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>4. Process individual Job Parts<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>5. Collate the individual results<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>6. Sumup/rollup the results<br />
<br />
In Short BigData processing is application of Parallel processing when data at hand is huge,unstructured and continuosly increasing, and hence cannot be processed using traditional single instance tools due to size and time limitations.<br />
<br />
When above <b>Divide and Conquer</b> template is used for grepping Billion lines, it would be as below.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>1. Setup 100 systems in network<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>2. Split the 1 Billion records into 100 files with each having 10,000,000(10 Million) Records<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>3. So, now we have 100 systems, and 100 files, copy 1 file to each system.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>4. Start grep on the files copied in step 3 on All 100 systems, store the result in temporary file.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>5. Copy the temporary files generated in step 4 to one central server<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>6. On central server, Merge all temporary files. And we have completed grep of 1 Billion lines.<br />
<br />
Ignoring the work/time overhead involved in splitting, copying , merging, we can say we completed the grepping of 1 Billion lines in under 3 hours! which would have taken 11 days on single system.<br />
<br />
Apart from these we also need to make sure of few things listed below.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>1. Since Data is huge in size, Our file system should support it and it should be accessible from all the systems in the network<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>2. Since our systems are not specialised and low cost, they are prone to failure, we should handle failure of individual systems and also make sure of no data loss.<br />
<br />
The above template can be used for processing data in many scenarios and not only limited to grep, Just need to follow the template of splitting the job, distributing, processing, and collating. To simplify and automate these steps, we have frameworks like Hadoop which automates most of the common steps like splitting, distributing, and collating of the results. Users of the framework need to work on only processing of the data.<br />
<br />
<b>More on Hadoop:</b><br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>1. It has distributed file system (called HDFS) which provides zero data loss environment for storing and retrieving TeraBytes of data<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>2. It has DataBases which use HDFS to store their data, hence provide a way to store structural data running to Billions / Trillions of rows<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>3. It has MapReduce frame work, Which can Divide Jobs, Distribute it, And Collate the result for final rollup/sumup. The MapReduce framework also handles individual system failure by keeping track of which job part is assigned to which system and reassigning it to different system on failures<br />
4. It has SQL like query engine named Hive to retrieve data from HDFS or Databases like Hbase. Hive internally schedules MapReduce Job to perform the task.<br />
<br />
Hadoop MapReduce is a batch/bulk data processing framework, You need to have complete data at hand to begin processing. This is not suited for many reallife scenario where we need realtime results. For example, Hadoop MapReduce can be great for finding out fraud in Banking systems by crunching billions of transactions done on previous day, But is not suited if the requirement is to stop a fraud in realtime, like stopping / blocking spammers abusing the system in realtime.<br />
<br />
Alternatively to Hadoop MapReduce, Frameworks like Twitter Storm allows processing of data in RealTime.<br />
<br />
<b>General Applications of BigData:</b><br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Most of the Applications of BigData is to derive insights, i.e Analytics of some form. General perception is BigData means Analytics and vice versa, However we can apply the above pattern for other usecases which are not analytics. One such non-analytical BigData deployment was done to main the huge mail system at my current company. Millions of mails needs to be archived/deleted/moved to different storage every day, Data at hand is huge(amounting to 10's of TB every day), This amount of data cannot be acted upon using traditional way, This has to be distributed for achieving operational targets, We applied above discussed pattern and achieved it using custom framework instead of using Hadoop MapReduce.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>BigData can also be used in system security, A Mail security system was deployed to keep a tab on compromised accounts and also to keep spammers/abusers in check, Again using custom framework. However, this security system uses some kind of analytics involving user's usage patterns/behaviour and hence would be categorised as "Analytics of some form".<br />
<br />
<br /></div>
Unknownnoreply@blogger.com1