Big Data Architecture - a walkthrough

Let me start this blog with a little example. Assume, Sachin have a leak in a water pipe in his garden. He takes a bucket and a some sealing material to fix the problem. After a while, he see s that the leak is much bigger that and he needs a specialist  to bring bigger tools. Meanwhile, he still uses the bucket to drain the water. After a while, he notices that a massive underground stream has opened and he needs to handle millions of liters of water every second. 

He does n't just need new buckets, but a completely new approach to looking at the problem just because the volume and velocity of water has grown. To prevent the town from flooding, maybe he needs his government to build a massive dam that requires an enormous civil engineering expertise and an elaborate control system. To make things worse, everywhere water is gushing out from nowhere and everyone is scared with the variety.

Welcome to Big Data.

Key elements of Big Data:

1. There are over 600 million tweets every day that is flowing every second which tells about the High Volume & Velocity
2. Next  need to understand what each tweet means - where is it from, what kind of a person is tweeting, is it trustworthy or not which tells about the High Variety
3. Identify the sentiment - is this person talking negative about iPhone or positive? which describes about the High Complexity
4. And finally need to have a way to quantify the sentiment and track it in real time which tells about High Variability

Traditional architecture of any Big Data solution would look something like below,

Data is collected from various sources such as Content management system, software application and then transferred to relational database management systems such as mssql,postgresql etc. 
In order to analyse these data which are being collected , ETL part is done using a single machine most of the times and the necessary data is being transfered to some of the OLAP data warehouses  for analyzing data. The data which is analysed finally is archived data which is not live/real time data. Which in terms refered as Death of data.  Data which is finally analysed is only 10%. 

Big Data,  is massive data that cannot be stored and processed by a single machine. Having understood the basics of Big Data, it is important to learn how it can prove helpful in enhancing business opportunities. Given the massive volume of Big Data, it is logical that only some of it would be of extreme importance. It is this small percentage of data which when analyzed and used in the right manner can prove quite advantageous for promoting business online. 

A modern Architecture:

I have transformed the above architecture inorder to support large volume of data with the modern tools and technologies, A Modern BI architecture will be cooked by using the recipe as below:

Am I really a developer or just a nethead?

It has been 6 years since i entered into programming field, and 18 years since i started using a computer, Everyone things i am a computer geek. Some times in my mind sounds come that Is that I am really a developer or just a good nethead?.

It's because there has not been a single day i coded without using Google search and Stack Overflow.

My Experience

When I was 15, I wrote my first program.  That was a long time ago now.  And it was in pascal language. During my university days i had more interest on gaming and animation rather than programming. When i got my first job, i struggled to code in c# with the dot net platform  during the initial days. With the help of google and stack overflow, now i would rate myself 8 on c# and i have experience of various open source technologies. But still i felt i was a better googler , not a good programmer. 

What made me to think I am a really bad programmer?

(i) Choosing workarounds over doing the right thing.

(ii)Used Ctr+c and Ctr+v more than normal keys

(iii)When things went wrong, i asked who is at fault rather than what the problem was. 

Mistakes to avoid to become a better programmer

In the year 2016, i started to avoid above practices and i would say programming is the first step to solving problems using technology. My tips that i followed during the year to be a better programmer as follows,

1. Every day find a small challenge that can  be done in an hour.
2. Read code. There is a plethora of freely available code for applications. There are tons of free projects by others on github.
3. Make small projects to build experience.  Make it an open source project and if you can encourage collaboration if your project is compelling enough.
4. Try programming for a day without googling. continue it for two days, maybe a week. See how it feels.
5. Go to Meetups, Workshops, meet with others who feel the same way you do about technology.

anyone can become a good developer if he/she is passioned about it and practice a lot, preferably daily."In order to remain at the same level you have to spend at least two hours daily programming.". There will be many programmers out there who would think the same! What do you think?