Saturday, 15 October 2011

MaxHeap's very own BI

With this, we plan on taking you through a brief tour of all the cool stuff that has happened with our tools team in recent times.

The eventual need of every company is business intelligence and a term synonymous with that is large volumes of data. Like a company, we generate hundreds of thousands of lines of logs a day. And since we call it a log, the data we record in there remains trapped for all eternity unless someone decides to do something about it.

So the tools team rose to the occasion and through our catalyst and founder, Lalit's suggestion, we decided to do something cool here. Lalit basically did an excellent job at analysing what a typical BI system would need. He was spot on about having room to twist malleable data, clean it, record it and use it as we'd want to. So we needed something that could do a little more than a normal data storage backend like a typical SQL database. He suggested MongoDB, and that is when I fell in love with this wonderful database.
We had been aiming at a simplistic system earlier, but as I dived deeper into Mongo and noSQL territory, the horizons started widening instantly, expansively. I instantly knew that a lot more was possible with this database – and that was the need of the hour; a system we could scale massively.
So I decided to go much further than a business analytics system. I decided to deliver a framework itself.

The initial phases of development were not taxing. The transition from SQL to noSQL was seamless. And within a couple of weeks, we were dealing with humongous amounts of data, parsing it, cleaning it and feeding it to the DB. Just like that.

We then proceeded to build a report scheduling system that again worked on top of mongoDB. Not just that, we pushed and implemented a caching system that could now be used not just by the BI framework but by anyone in any team in the need of some really fast cache on the web.

All of this didn't need genius coding. It only needed the tools team and mongodb.

27 comments:

  1. I am not sure of the details you are divulging here , but given the logs , you can implement something like a failure alarm or a tool that allows you to select the hosts you want to search logs of , which can spit out the errors that have happened . That was about the OLTP data.
    If you already have a reporting system in place , you can setup an OLAP environment which can accumulate the events over time and generate dashboards that the you can view for their analysis.

    ReplyDelete
  2. Hello Harshit. What you've pointed out, we already have in place. A business intelligence framework is very different. And the idea behind using mongoDB to store and mine the data from solves all the requirements an OLAP solution would.
    In any case, you'll need to be able to answer 'questions' from the past which is exactly what we've aimed to solve - and to a huge extent, we've been successful at that.

    ReplyDelete
  3. Great Article
    Cloud Computing Projects


    Networking Projects

    Final Year Projects for CSE


    JavaScript Training in Chennai

    JavaScript Training in Chennai

    The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

    ReplyDelete
  4. The market feasibility of population statistics and psychological statistics is studied. Usually, there is a huge demand for uesed cars for sale
    . Several surveys over the years have shown that people who can afford new cars are much less than those who like to buy second-hand cars.

    ReplyDelete
  5. https://www.bbdd66.com 우리카지노사이트
    https://www.bbdd66.com/theking 더킹카지노
    https://www.bbdd66.com/sands 샌즈카지노
    https://www.bbdd66.com/first 퍼스트카지노
    https://www.bbdd66.com/yes 예스카지노
    https://www.bbdd66.com/super 슈퍼카지노
    https://www.bbdd66.com/gatsby 개츠비카지노
    https://www.bbdd66.com/33 33카지노
    https://www.bbdd66.com/world 월드카지노
    https://www.bbdd66.com/merit 메리트카지노

    ReplyDelete
  6. https://www.oobbg.com 우리카지노사이트
    https://www.oobbg.com/theking 더킹카지노
    https://www.oobbg.com/sands 샌즈카지노
    https://www.oobbg.com/first 퍼스트카지노
    https://www.oobbg.com/yes 예스카지노
    https://www.oobbg.com/coin 코인카지노
    https://www.oobbg.com/33 33카지노
    https://www.oobbg.com/world 월드카지노
    https://www.oobbg.com/merit 메리트카지노
    https://www.oobbg.com/gatsby 개츠비카지노

    ReplyDelete
  7. https://www.omgka.com 우리카지노사이트
    https://www.omgka.com/theking 더킹카지노
    https://www.omgka.com/sands 샌즈카지노
    https://www.omgka.com/first 퍼스트카지노
    https://www.omgka.com/yes 예스카지노
    https://www.omgka.com/super 슈퍼카지노
    https://www.omgka.com/gatsby 개츠비카지노
    https://www.omgka.com/33 33카지노
    https://www.omgka.com/world 월드카지노
    https://www.omgka.com/merit 메리트카지노

    ReplyDelete
  8. https://www.omgqq.com 우리카지노사이트
    https://www.omgqq.com/thekingcasino 더킹카지노
    https://www.omgqq.com/sandscasino 샌즈카지노
    https://www.omgqq.com/firstcasino 퍼스트카지노
    https://www.omgqq.com/yescasino 예스카지노
    https://www.omgqq.com/supercasino 슈퍼카지노
    https://www.omgqq.com/gatsbycasino 개츠비카지노
    https://www.omgqq.com/33casino 33카지노
    https://www.omgqq.com/worldcasino 월드카지노
    https://www.omgqq.com/merit 메리트카지노

    ReplyDelete
  9. https://www.omgab.com 우리카지노사이트
    https://www.omgab.com/theking 더킹카지노
    https://www.omgab.com/sands 샌즈카지노
    https://www.omgab.com/first 퍼스트카지노
    https://www.omgab.com/yes 예스카지노
    https://www.omgab.com/super 슈퍼카지노
    https://www.omgab.com/gatsby 개츠비카지노
    https://www.omgab.com/33 33카지노
    https://www.omgab.com/world 월드카지노
    https://www.omgab.com/merit 메리트카지노

    ReplyDelete
  10. Managing the databases is very important because companies running on big data deal with databases with care. Like when I used APA reference generator for my research paper, I wondered that how much data these tools hold and deliver when needed. This is the best example of databases.

    ReplyDelete
  11. شكرا لك على نشر مثل هذه المعلومات الرائعة يلا لايف

    ReplyDelete

  12. kora online
    شكرا لك على نشر مثل هذه المعلومات الرائعةكورة اون لاين

    ReplyDelete
  13. Every company has a centralized database and database administrator that maintains the database quality and overall functionality. I am working as an academic writer and provides
    dissertation writing service UK.Data is an important asset for the company. Therefore, managing a database is mandatory.

    ReplyDelete
  14. مشاهدة مباريات اليوم بث مباشر
    bein match

    ReplyDelete
  15. مشاهدة مباريات اليوم بث مباشر
    بدون تقطيع
    بين ماتش

    ReplyDelete
  16. مشاهدة مباريات اليوم بث مباشر
    beinmatch

    ReplyDelete
  17. شاهد اهم القنوات العربية بث مباشر
    قنوات مصرية

    قنوات mbc

    ReplyDelete
  18. I think I am literally the only person in New Zealand who watches the IIHF World Juniors 2022 Streaming. Then again, I also watch our U20s play in Div 3, and the National Team play in Div 2B, so I'm well outside the normal parameters of fandom.

    ReplyDelete
  19. This is Couponplay, a professional blogger I enjoy writing on multiple callings with a firm grip on specialty include digital marketing, affiliate marketing, entertainment, and e-commerce blogs.

    ReplyDelete
  20. I spent a long time looking for a woman who would support and assist me. But, sadly, I couldn't locate such a girl in my nation. It was a period when I really needed a woman's help. Then I stumbled upon an article about ukrainian women for marriage. I was intrigued by the prospect, so I started looking for it online. I didn't get very far, but you may find a woman who will love and support you with these guys. I wish to suggest these gentlemen to everyone since they will not disappoint you and may perhaps alter your life.

    ReplyDelete
  21. Hey would you mind stating which blog platform you’re using?
    I’m looking to start my own blog soon but I’m having a hard time deciding between BlogEngine/WordPress/B2evolution and Drupal. do my homework for me
    The reason I ask is because your layout seems different then most blogs and I’m looking for something unique. P.S Apologies for
    getting off-topic but I had to ask!

    ReplyDelete
  22. I have read it completely and got really impressed. Love to read more about similar types of articles Like Dissertation Writing Services . Thanks a lot.

    ReplyDelete