Kafka Topic for Web Application Data

TLDR

Share data held in the "web application" domain with other services via the means of a Kafka topic.

My Role

Tech lead of the team that implemented this feature.

Context

Several systems in Employee Referrals (ER) require data from the webapp domain for functioning. To address this, we set up a Kafka topic to hold the entire state of a tenant's web application config, that other services could consume from.

Challenges

• The biggest challenge was to come up with a strategy to push messages to the Kafka topic. We decided to use the "outbox pattern" for this. An outbox table in the database stores entries corresponding to changes in the attributes we're concerned with. Producers poll this table, construct messages, and push them to the Kafka topic.