Ad-click aggregation

Why do the click counts never match the kitchen’s?

The twenty partner kitchens from lesson 5.8 can now pay to promote their menus in the app. Each tap on a promoted menu costs the kitchen a fixed fee, and the monthly bill is broken down minute by minute through the lunch rush.

The counter reads the tap events from the order log. It files each tap under the minute in which it happened, using the time the phone stamped on it. It closes a minute once it has seen a tap stamped five seconds or more after that minute ended. Any tap that arrives for a minute that is already closed is thrown away.

The largest partner sent back its own log of visitors from the app, minute by minute. Between 12:00 and 12:10 the bill said 411 taps. The partner counted 422. The totals are close, but the partner compared every minute, and eight of the ten minutes were wrong: some too high, some too low.

Two causes pull in opposite directions. Customers commuting by train tap the app in the ghat tunnels; the phone keeps the tap and sends it minutes later, after its minute has closed, so 28 taps were dropped. And when an acknowledgement is lost, the phone sends the same tap again, so 17 taps were billed twice. Do the arithmetic: 422 minus 28 plus 17 is 411. The two errors nearly cancel in the total, which is why nobody noticed for a month.

Kafka’s exactly-once processing sounds like the fix. It stops one log record from being counted twice, but a tap resent by the phone is a second record, and no delivery guarantee says which minute a late tap belongs to. Two questions remain: how long should a minute stay open before its count is final, and how can the counter tell that two records are the same tap? When events can arrive late or twice, you have to choose between a count that is fast and a count that is final.

Event time, watermarks and lateness

Event time and processing time

Event time is when something happened, stamped where it happened. Processing time is the clock of the machine handling it. Bills, contracts and partners care about event time.

For example: A tap at 12:04:10 in a tunnel that reaches the server at 12:08:30 has event time 12:04 and processing time 12:08.

Watermark

The counter’s running claim: “I believe no more events older than time t will arrive.” When the watermark passes the end of a window, the window fires its result.

For example: The newest tap seen is stamped 12:05:07 and the lag is 5 s, so the watermark is 12:05:02: the 12:04 minute fires.

Two more settings decide what happens to a tap that arrives after the watermark has passed its minute. Allowed lateness keeps a window’s state for a while after it fires; a late tap within that time fires the window again with a corrected count (Flink’s default allowed lateness is 0). After that, the state is deleted and a late tap is dropped, unless you route it to a side output to be settled some other way. Separately, a tap delivered twice is two records with one click id: only a check on that id stops it being billed twice.

Fast numbers or final numbers?

A dashboard for the lunch rush wants the counts now. A bill wants them right. Both read the same stream.

The dashboard and the bill are different products of the same stream, so they can have different deadlines. The cost is real: windows and click ids kept in memory for minutes longer, and every consumer of the stream must understand that an early count can still change.

What to remember

1. Count by event time when the number is about when things happened; processing time is cheap and wrong whenever delivery is delayed.
2. A watermark is a guess that a window is complete. Too short and data is late; too long and every answer waits.
3. Allowed lateness lets a fast first answer be corrected; a side output keeps what is later still, for a reconciliation.
4. Duplicates are a separate problem from lateness: dedupe by an id that the sender keeps when it retries.

Count by the time it happened, and decide in advance how long a minute stays open.

In the real world

Real click pipelines add what the toy leaves out: fraud and bot filtering before anything is billed, dedupe stores that forget ids after the lateness horizon, watermarks per source merged across partitions, and idle sources that would otherwise hold the watermark back. Many also keep the raw event log and recompute bills in batch, using the streaming count only for dashboards. That batch pass is the same idea as reconciling against the outside world in lesson 6.7.