kafka usecase in real world

movieflix

image.png

• topic:

video play position:
  a topic that can have multiple producers;
  should be highly distributed if high volume > 30 partitions;
  choose "user_id" as a key, to make sure all my user's data in order;

recommendations
  low volume topic
  kafka streams recommendation source data from analytical training

GetTaxi

requirement

image.png

social media architecture of kafka

basically decoupled different components, we can call the model CQRS(command query responsibility segregated)

image.png

• format data as an event.

User_234 liked post_123 at 3 am

image.png

---

• big data ingestion

image.png

real time: like spark/storm
batch: hadoop/RBMS

---

monitoring

image.png

important metrics

---

two types of index:

image.png

---

image.png

---

advertised hostname and listeners:

image.png