kapil garg

  Today i watched one video on kafka consumers presented by Igor Buzatoic at kafka summit. He demonstrated the working of kafka consumers, group re-balancing and multi-threading environment. The complete video should become available on youtube soon. So the presentation was divided basically in 2 sections - Single threaded mechanism - Multi threaded mechanism Single Thread The idea here is that a single thread is responsible to poll the records from kafka and do the processing of those records. Now here the offset commit step can either be executed manually after the processing is done or periodically if we are sure than our delay in processing is less than commit interval. Usually people go with manual commit only as this ensures that we are committing what we have already processed. Now this is simple architecture and doesn't require any synchronization. Multi Thread In multi threaded environment, there are 2 possibilities - Single thread responsible for polling the records, proc...

Hi, I am kapil garg and i have created this blog to post anything i learn or find interesting in day to day life. You may find something to learn here as well. I am currently an employee at Flipkart Pvt. Ltd. working as a software engineer. Most of my work revolves around huge amount of data processing and event streaming. I take help of Apache Spark and Apache Kafka to handle the data and all the events that come my way. I don't intend to teach anything in this blog so you might find my posts a little rough but that's okay. If you wish to only learn something then i would suggest you to checkout my other blog fetch-info. I have also worked in Samsung Semiconducter as a software engineer and there i worked mostly on database designs and SSD frameworks.   Currently i am focusing on Apache kafka and i will try to contribute to the project. Also there are some personal projects going on which are not noteworthy right now but if i learn something doing those, i will post it. I g...