The eventual need of every company is business intelligence and a term synonymous with that is large volumes of data. Like a company, we generate hundreds of thousands of lines of logs a day. And since we call it a log, the data we record in there remains trapped for all eternity unless someone decides to do something about it.
So the tools team rose to the occasion and through our catalyst and founder, Lalit's suggestion, we decided to do something cool here. Lalit basically did an excellent job at analysing what a typical BI system would need. He was spot on about having room to twist malleable data, clean it, record it and use it as we'd want to. So we needed something that could do a little more than a normal data storage backend like a typical SQL database. He suggested MongoDB, and that is when I fell in love with this wonderful database.We had been aiming at a simplistic system earlier, but as I dived deeper into Mongo and noSQL territory, the horizons started widening instantly, expansively. I instantly knew that a lot more was possible with this database – and that was the need of the hour; a system we could scale massively.
So I decided to go much further than a business analytics system. I decided to deliver a framework itself.
The initial phases of development were not taxing. The transition from SQL to noSQL was seamless. And within a couple of weeks, we were dealing with humongous amounts of data, parsing it, cleaning it and feeding it to the DB. Just like that.
We then proceeded to build a report scheduling system that again worked on top of mongoDB. Not just that, we pushed and implemented a caching system that could now be used not just by the BI framework but by anyone in any team in the need of some really fast cache on the web.
All of this didn't need genius coding. It only needed the tools team and mongodb.