How to hire a big data engineer, with Qubole co-founder, Joydeep Sen Sarma (also one half of the team behind Apache Hive)

Wikipedia defines big data as the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. Big data engineers work to solve the problems of capture, curation, storage, search, sharing, transfer, analysis and visualisation of data.
But everyone, from engineers, to entrepreneurs, to investors to marketers are talking about it as if it were the next big thing. But rest assured, more than 90% of them wouldn’t know what it means. It’s become a fad. So much so, that behavioural economics guru, Dan Ariely said this about big data -
“Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.”
The overuse of this phrase becomes particularly worrying when you look at the job descriptions for big data engineers. Adjectives are thrown around carelessly, giving the applicant everything
So what exactly should you be looking for when you’re looking for a big data engineer? Speak to enough people who have actually worked on big data projects, and they’ll tell you to look for a deep understanding of stats and regressions, correlations, Bayseian modelling and probability theory. Along with this, you’re also looking for the usual technology skills – Good with coding, and writing programs for scale. Big data engineers should also not be confused with data scientists. In fact, most of the time, they’re working with comparatively smaller data sets.
I had the chance to speak with Joydeep Sen Sarma, co-founder of Qubole, and one the founders of Apache Hive, the popular data warehouse infrastructure built on top of Hadoop, on what he thought about hiring big data engineers. He said, “In my honest opinion, that’s a sort of meaningless question. There’s no such category called big data. A lot of people who work on data processing backends are just regular programmers with backend expertise.”
“If u are talking about people who put things in production – then familiarity with the basic tooling that has emerged in this space helps. It is standard list of software. That would be the closest to ‘big-data engineer’ I can think of.”
I further asked him what a company is actually looking for when they say they want to hire a big data engineer, he says, “Typically they are looking for people who can implement big data projects collecting logs, designing large scale warehouses, building backends for real-time and batch reporting, integrating different backend systems and so on. These are people who work with big data tooling. They are in great demand. Very few of them have to be expert in data science or actually writing code for large scale distributed systems. Their value lies in understanding and mastery of a complex and emerging basket of tools used for manipulating large data sets”
So the next time you’re browsing a job site, don’t be deterred by the phrase big data. You might be more than a fit for the role. And on the part of the company’s, the term big data in the context of engineering is very ambiguous.
10 highest paying IT companies in India
Globalization has changed fortunes for Indian Economy in last 20 years and none other than India’s IT industry has developed from virtually nothing to an industry of $200 billion in last 2 decades. Every new thing brings in new opportunities with it. But what does it meant for us? What has this change done for us in terms of employement? Everyone thinks IT employees as one of the highest paid employees in any kind Industry in India today. Having amazing salaries, perks, cool technology ,swanky offices and etc. etc. makes their jobs best among a lot.
But there are always some top leaders in every field and nothing is different in IT sector as well. Here is a list of top 10 IT companies which pays most in IT field and some interesting stats related to them. We have actually
| Companies | Software Engineer(in LPA) | Senior Software Engineer (in LPA) | IT Project Manager(in LPA) | ||||||
| Start | End | Median | Start | End | Median | Start | End | Median | |
13
|
29
|
14
|
15
|
30
|
26
|
19
|
36
|
29
| |
| HP |
3
|
12
|
7
|
6
|
14
|
10
|
7
|
23
|
15
|
| IBM |
3
|
14
|
8
|
5
|
17
|
10
|
7
|
20
|
16
|
| Microsoft |
11
|
21
|
12
|
14
|
28
|
18
|
11
|
22
|
32
|
| Juniper systems |
5
|
22
|
14
|
6
|
26
|
17
|
22
|
31
|
26
|
| Goldman Sachs |
5
|
23
|
10
|
7
|
25
|
16
|
26
|
28
|
27
|
| Adobe |
6
|
20
|
9
|
6
|
21
|
13
|
25
|
36
|
31
|
| CISCO |
6
|
25
|
9
|
7
|
22
|
13
|
11
|
25
|
18
|
| Intel |
5
|
21
|
11
|
8
|
21
|
13
|
21
|
24
|
22
|
| Amazon |
5
|
19
|
11
|
6
|
21
|
13
|
30
|
34
|
31
|
Google
It has come to rescue for virtually each and every Indian college student at the time of their projects, presentations and whatever it may be. This silicon valley based company supported by an Indian named Rajeev Motwani in its early days, is one of the highest paying IT company in India at present. Giving a median salary of around 14 lpa for freshers in India plus the added brand value of working with this company is nothing less than finding a gold mine for a fresher in India. Average salary of a senior software engineer in Google India is more than 25 lpa.
HP
Hewlett-Packard, populary known as HP was founded in a one car garage. HP has total assets of more than $100 billion and a team consisting of more than 317,500 employees today spreaded all around the globe. HP provides an average of 7.5 lpa for freshers as software engineer and the same increases to around 1 million in an interval of 5-6 years. Salary of a software developer at HP is around 6.5 lpa which is 19% more than what software developers get in Indian market.
IBM
In making IBM a company holding the most no. Of patents for 20 years now, we Indians have played a major role. In return, IBM has shown their grace by recruiting highest no. of Indians by any IT MNC in India at present. An average of more than 8 lpa for software engineer and 11 lpa for senior SE’s is what IBM offers. But as job security is considerably less in this company so it is not among the most employee friendly IT companies in the market.
Microsoft
Microsoft mounted to rule the home computer operating system market with MS-DOS in the mid-1980s, followed by the Microsoft Windows line of operating systems. An average salary of 11.5 lpa for a fresher and 18 lpa for a senior engineer in India which is way more than the average salaries for the same job in India at other companies, plus the brand value of microsoft is always there for you.
Juniper Systems
Founded by an IIT Kanpur alumni, Juniper is an america based company which is a manufacturer of networking systems. Average salary of Juniper systems in India is around 14 lpa which is even more than the IT giants google and microsoft. Juniper provides 30% above the market when it comes to salary of Senior software engineers and developers.
Goldman Sachs
Being an investment company and still giving highest average salary package for software engineers in India is really surprising. An average salary of more than 10 lpa per annum for a software engineer and more tha 15 lpa for a senior SE, GS is what you are looking for if you want to earn money. But be sure about it, because 15 lpa on an average is not for free, Investment banks always know how to take best out in what they have invested 
Adobe
More than an average salary of 9 lpa for freshers as Software Engineer and 8 lpa for software developer is one of the best salary packages at this time around in India. Alongwith this growth as an individual is a very big additional perk in Adobe. The salary hike is almost 1.5 times the starting salary in a time of around 6 years. Median salary for a senior software engineer in adobe is more than 13 lpa.
CISCO
With more than 60,000 employees Cisco’s annual revenue is $ 43.0 billion as of 2011 and is one of the leading IT companies in the world. CISCO provides an average salary of more than 9 lpa for the freshie Engineers and 6.5 lacs for software developers which jumps to 13 and 8.6 lpa respectively in 5 years time.
Intel
The average salary of employees at Intel is around 10 lpa. The company is considered to be a paymaster when it comes to salaries for its employees. Good and hardworking employees enjoy salary packages of around 10 to 11 lpa. The growth for an employer is also good considering the change in median salary from a software engineer to senior software engineer is pretty good.
Amazon
An E-commerce giant, it might still be finding its feet in Indian market but the people who have already find their feet in this company are totally going to be money maker. Amazon provides a median salary of 10 lpa for software engineers having around 2 years of experience and 13 lpa to senior software engineer. Average salary of around 7.5 lpa for a fresher software developer is one of the best in India for this profile.
It may come as a surprise for some persons that no Indian IT company is in the list. TCS and Infosys are the leading Indian tech companies which provides maximum average salary for SE and SD posts which is around 6-7 lpa. TCS has also just broke into top 10 IT companies of the world in terms of its revenue and is continuing with its the biggest bunch of IT employees in India. Other Indian companies following TCS and Infosys are Wipro, Mahindra British Telecom and L and T infotech.