Apache Kafka for Event-Driven Microservices with Laravel and Node.js

Apache Kafka is a durable log that many services can read, independently, at their own pace. That is the whole trick. People wrap it in the word “microservices” and then build a distributed monolith that still needs a lock on a MySQL row. If you are a CTO hiring an agency, you want Kafka when several consumers must react to the same business fact without sharing a database, and when you need to replay. You do not want it because a diagram looked senior.

DayDecoder’s founder works with Kafka on real products; we implement event-driven slices in Laravel and Node.js from Ahmedabad for overseas teams. This is a practical architecture guide, not a certification dump.

When Kafka is the right tool

Facts, not “UserService please update”

A good event names something that already happened, with a stable id, a time, a tenant, and a payload version. Consumers you have not written yet should still be able to read it. If the payload is “call this HTTP URL”, you have built a brittle RPC.

Schema: JSON plus a documented version is fine to start; Avro/Protobuf plus a registry when many teams share the log. Breaking changes are new versions, not silent field reuse. Include tenant id on every message if you are multi-tenant. Treat the log as a place other tenants must never appear, same as a SQL WHERE tenant_id.

Laravel as a citizen

Laravel will not be the cluster. It will produce from a domain action after the transaction commits (outbox pattern) and consume from a long-running worker, not from a web request. The outbox is how you avoid “HTTP 200, event never published” and “event published, DB rolled back”.

Practical pieces:

• An outbox table written in the same DB transaction as the business row.
• A publisher daemon that reads the outbox and writes to Kafka, then marks published.
• A consumer worker (RoadRunner, a dedicated process, or a well-supervised PHP worker) that processes a batch, writes side effects idempotently, then commits offsets.
• Horizon-style ops for the worker: restarts, lag alerts, a dead-letter topic.

Do not Kafka::send() in a controller and hope. Do not run consumers on PHP-FPM.

Idempotency: store processed event ids (or use the Kafka offset plus partition as a key) so a rebalance does not double-fulfil an order. If you charge cards, this is not optional.

Node.js as a citizen

Node is a natural Kafka client: long-lived process, good libraries, streaming. The same rules: do not process in an Express request handler; commit after side effects; bound memory; handle rebalance. Type the payload. If you also have a Laravel monolith, Node might own a specialist consumer (search index, websocket fan-out) while Laravel owns checkout. That is a boundary, not a fashion.

Topics, partitions, and keys

Key by the entity you need order for (order id, device id). Partition count is an ops decision you will live with; changing it later reshuffles. Do not make a topic per customer. Compacted topics are for tables-as-logs (current state of a device), not for every click.

Retention is money and compliance. “Forever” is a GDPR conversation, not a default. Know who can read a topic. ACLs are part of the build.

Failure, poison, and replay

A message that always crashes a consumer will block a partition if you naively retry forever. Retry with a bounded backoff, then a dead-letter topic with the error and the original payload. Humans inspect. Replay is a button with a time range, not a hope. This is where Kafka earns its keep compared with a fire-and-forget queue — if you actually operate it.

Exactly-once across Kafka and MySQL is a design (transactional outbox + idempotent consumer), not a config flag you tick in a UI and forget.

Local and production

Docker Compose for a broker is fine for developers. Production wants disks, monitoring of under-replicated partitions, consumer lag, disk usage, and a paging policy. Managed Kafka is often the right buy so your team writes consumers, not JVM tuning. That buy-vs-build is the same honesty as custom vs vendor, applied to infrastructure.

A delivery sequence we trust

1. One business fact, one topic, one consumer, logs you can grep.
2. Outbox on the producer, idempotency on the consumer.
3. Lag dashboards and a dead letter.
4. A second consumer on the same topic (the actual reason you bought a log).
5. Only then more services. If you cannot operate one topic, you cannot operate twenty.

DayDecoder has 10+ years building product software for 103 customers, including event-driven work. We will tell you if you should stay on Redis queues. Email contact@daydecoder.com.

Transactional outbox, in enough detail to implement

Write the business row and the outbox row in one database transaction. A publisher process (not the HTTP worker) reads unpublished rows, sends to Kafka, then marks them sent with the broker offset or a producer id you store. If send succeeds and the mark fails, you will retry and the consumer must be idempotent — that is expected. If you publish before commit, consumers will see orders that rolled back. That is the classic bug. We look for it in code review before we look at topic names.

For Node, the same pattern with a SQL outbox, or a dual-write you accept is weaker. Dual-write (DB then Kafka in two calls) will eventually drift. If you start that way, put a reconciler on the calendar, not on a wish list.

Contracts between Laravel and Node

If checkout is Laravel and search is Node, the event is the contract. Version it. Publish a short JSON schema in the repo both sides can CI against. Do not rely on “we’ll Slack the change”. A consumer that ignores unknown fields survives longer than one that explodes on an extra key. Additive changes first; breaking changes as a new event type.

End-to-end tests on a dockerised broker are worth it for the checkout path. You do not need to test every topic that way. You need the money path.

Security of the log

ACLs per topic, encryption in transit, encryption at rest if the vendor offers it, and no PII you do not need. A log that retains a year’s worth of emails is a disclosure waiting for a lawyer. Mask. Hash. Drop. If you need a payload for replay, store a pointer to the object store and a retention job. Kafka is not your data warehouse, even though people query it like one.

Admin tools (AKHQ, Confluent UI, and similar) are privileged. SSO. No shared admin on a sticky note.

Cost and sizing without fake benchmarks

You pay for storage, network, and brokers. Fat events and long retention dominate. Compact when you store state. Do not log a 2 MB JSON of the entire user on every click. We will not quote a “typical Kafka bill”; it depends. We will look at payload size and events per day on a call and tell you if a managed starter is enough or if you should stay on Redis until the second consumer exists.

A useful internal metric: consumer lag in seconds for the payment topic, with a page. Lag in messages without time is how you fool yourself at low traffic.

Naming, headers and trace ids

Put a trace id on the HTTP request that created the outbox row and copy it into the event headers. Then every consumer log can be grepped. This is cheap and makes Kafka feel like a product instead of a fog. Message keys stay the business key; trace ids stay in headers. Do not key by trace or you will break ordering per entity.

Header conventions belong in the same repo as the schema. Laravel and Node should use the same names. “We used camelCase they used snake_case” is a real outage report. Pick one, write it down, lint it if you can.

When you must fan a single fact to a search index, a warehouse, and a websocket, that is three consumers, one topic, three idempotency stores. Resist a “universal consumer” that does all three in one process. That process will become a monolith with a Kafka logo.

Local development should not require a full production cluster. One broker in Compose, a seed topic, and a Makefile target that creates topics with the right partition count is enough. Document the commands in the README so the next hire does not invent a second topic naming scheme. Production topic names should match local names plus a prefix, not a poetic rename. Drift in names is how events go nowhere and nobody notices until finance asks where the invoices went.

When you delete a service, delete or drain its consumer group and its ACLs. Abandoned groups look like lag ghosts. Abandoned ACLs look like surprise access. Kafka ops is gardening. Budget a little weeding in the same retainer you would budget for Horizon failed jobs.

Handover checklist

A Kafka slice is handed over when you have: topic names and partition counts, schema (or JSON examples) in git, outbox table documented, consumer group ids, lag alert, dead-letter procedure, ACL list, and a Compose file a new developer can start. Add the trace-id header name. Add who may run a replay and from which environment. Replay from production into a laptop is a data incident, not a debug flex. Write the forbidden replay paths down. Include how to add a second consumer without changing the producer. That is the test that you bought a log, not an HTTP detour. If that sentence is hard to demonstrate, you are not done, even if the first consumer works on a happy path in staging.

FAQ

Kafka vs RabbitMQ vs Redis streams?

Kafka is a replayable log with independent consumers. Rabbit and Redis streams are often simpler queues. If you do not need replay and many independent consumers, do not start with Kafka. We will pick the boring tool that matches the job.

Can PHP really consume Kafka?

Yes, with a long-running process and a solid client. It is the wrong language for the broker itself. If your team is stronger in Node for the worker, that is a valid split.

Do we need microservices to use Kafka?

No. A modular monolith can publish facts and grow consumers later. Microservices without a log often share a database anyway, which is the coupling you were trying to avoid.

What about schema registry?

Add it when more than one team produces to the same topics, or when you are tired of undocumented JSON. It is not day-one mandatory for a single product team with a wiki.

Will Kafka lower our AWS bill?

Usually the opposite at small scale. You buy capability. Cost control is partition count, retention, and not duplicating the same payload to twenty fat topics. See our cloud cost work if the bill is the actual pain.