Processing Vehicle Telemetry with Go

How channels, batching, and back-pressure help us process a continuous stream of GPS and vehicle data without overwhelming the systems around it.

By Pratik Jain

A single vehicle can send a GPS update every few seconds, along with speed, battery, and heartbeat data. Handling one vehicle is easy. Across the fleet, this becomes a write-heavy stream of around 15,000 GPS location rows per minute.

The most direct implementation looks reasonable:

consume one event → decode it → insert it → publish it → consume the next

It also works well in development, where traffic is low and the database is nearby. Under sustained load, however, each step starts slowing down the next one.

Database writes block ingestion. If every event waits for its own insert, a slow database call delays the next Kafka poll and consumer lag grows.
One insert per event creates unnecessary overhead. The database spends time on repeated network calls, statements, and transactions instead of processing efficient sets of rows.
A goroutine per event only moves the queue. Messages leave Kafka quickly, but now thousands of goroutines compete for memory and the same database connection pool.
Publishing every GPS point spreads the problem. Downstream services receive the full input rate even when they only need a recent location.

The issue was not simply how to consume messages faster. We needed a design that could:

Absorb short traffic bursts without using unbounded memory.
Slow ingestion safely when processing fell behind.
Write historical data efficiently while keeping current state easy to read.
Send useful live updates without forwarding every GPS point.

We built that ingestion path in Go, using Kafka as the input stream, PostgreSQL and TimescaleDB for storage, and RabbitMQ for selected live updates.

This post follows one GPS event through the pipeline and explains why we chose bounded channels, worker goroutines, batching, and controlled fan-out.

The pipeline

Vehicle telemetry ingestion pipeline

The flow has two goroutines:

A poller reads messages from Kafka and sends them to a channel.
A worker reads the channel, decodes each message, sends selected live updates, and adds records to a batch.

We use the same basic shape for other telemetry streams. Their outputs can differ: GPS needs history, current location, and live fan-out, while another stream may only need the latest state.

The reusable idea is the boundary between reading and processing—not identical storage for every event type.

A channel is a buffer, not extra capacity

Without a channel, polling and processing are tied together:

poll → decode → database work → poll again

If a database operation is slow, the next Kafka poll is delayed. A channel lets polling and processing overlap:

poll → channel → poll
↓
worker → batch

It absorbs short bursts

Traffic is rarely steady. A buffered channel lets the poller move briefly ahead while the worker catches up.

The channel must be bounded. An unlimited in-memory queue can hide an overloaded service until memory is exhausted.

It creates backpressure

Sending to a full Go channel blocks. When the worker falls behind, the poller slows down and the remaining records stay in Kafka instead of accumulating forever in process memory.

That is backpressure: the slower stage controls the faster one.

A larger channel does not increase long-term throughput. If the incoming rate stays above the processing rate, any finite buffer eventually fills and Kafka lag grows.

A bounded channel absorbs temporary imbalance. It does not fix a permanent capacity gap.

Why not use one goroutine per event?

Goroutines are lightweight. Database connections, memory, and external systems are not.

Starting a goroutine for every event would allow a burst to create many database operations at once. The database connection pool would become the real queue, memory would grow with blocked goroutines, and ordering would become harder to reason about.

We instead use bounded concurrency: each telemetry stream has a controlled processing path. If one worker is not enough, we can scale deliberately by:

Adding Kafka partitions and consumer instances
Using a fixed-size worker pool
Partitioning by vehicle when per-vehicle ordering matters

The important point is that traffic volume should not decide how many operations run concurrently.

Batching is where the database savings happen

The channel controls flow, but it does not reduce database work. Batching does.

Our worker collects decoded events and flushes them when either a size limit or a time limit is reached. For GPS, a flush creates two set-based operations:

Append a batch of historical locations
Update the latest coordinates for the affected vehicles

many GPS events
→ one batched history write
→ one batched latest-state update

The database still processes every row. The improvement comes from fewer network round trips, fewer statements, less transaction overhead, and lower pressure on the connection pool.

The latest-state update also keeps only the newest value for each vehicle in the batch. This avoids writing intermediate values that are already outdated.

Batch size is a tradeoff

Larger batches usually improve write efficiency but keep data in memory longer. Smaller batches reduce delay but create more database operations.

There is no universal value. Batch limits should be chosen using:

Event rate
Acceptable storage delay
Record size
Database capacity
Recovery requirements

A time-based flush should use a real timer. Otherwise, a partial batch may remain in memory when traffic becomes quiet. Shutdown should also stop polling, drain the channel, flush the final batch, and only then close connections.

One GPS event, two different needs

A GPS event feeds two paths with different goals.

History and current state

The storage path keeps eligible points in a time-series store and updates a smaller latest-state store.

They answer different questions:

History: Where was this vehicle during a time range?
Latest state: Where was this vehicle last seen?

Keeping current state separately avoids searching a large historical dataset for every current-location request.

Live updates

Many downstream systems do not need every GPS point. Sending the full stream would increase message traffic and force each consumer to discard data it did not need.

We therefore publish live locations at a controlled rate per vehicle. A mutex-protected map records when each vehicle was last published and the timestamp of that location. A new update is sent only after the configured interval and only if it is newer.

The publisher is asynchronous: the GPS worker queues the update, while another goroutine performs the network operation. This keeps message-broker latency out of the worker’s critical path.

This split gives each destination the service level it needs:

Higher-resolution data for history
A compact value for current state
A lower-rate stream for live consumers

Storing useful detail does not mean sending every detail everywhere.

Why TimescaleDB for history?

Vehicle locations are time-series data. Most queries combine an entity with time:

Show a vehicle’s route during a journey.
Find the latest point before an event.
Compare the first and last points in a window.

TimescaleDB keeps PostgreSQL’s SQL model while adding features for this access pattern.

A hypertable divides records into time-based chunks, so a time-bounded query can avoid unrelated data. Device-oriented indexes help queries find one vehicle’s recent points. Older chunks can be compressed, and retention policies can remove data after the required period.

The time-series store answers “what happened over time?” The latest-state store answers “what is true now?” Keeping both models makes each query simpler and cheaper.

Kafka lag is not the same as completed work

Adding a channel introduces a second queue inside the process:

Kafka contains records that have not been polled.
The channel and batch contain records that may not be stored yet.

With automatic offset handling, Kafka can mark progress after a record is polled but before the database write finishes. Broker lag may look healthy while work is still waiting in memory.

If the process stops after the offset advances but before storage succeeds, that event may not be replayed. Stronger replay guarantees require offsets to become eligible for commit only after durable processing.

With batched writes, this means retaining topic and partition information and advancing only to the highest contiguous offset safely stored for each partition.

Monitor processing progress, not only polling progress.

Useful signals include channel depth, batch age, database failures, and the age of the newest stored event alongside Kafka lag.

Reliability must be designed explicitly

Channels and go routines improve flow. They do not decide what happens during failure.

A reliable pipeline needs clear answers to four questions:

When is an event safe to acknowledge? Ideally, after the required durable writes succeed.
What happens when storage fails? Retry, replay, or dead-letter handling must receive the error.
Can history and current state disagree? Separate writes need a transaction or a repair strategy.
What happens during shutdown? The process must stop intake, drain work, and flush partial batches in order.

These decisions are part of the architecture, not cleanup work to add later.

Written by Pratik Jain. Connect on LinkedIn.