# Architecting a Real-time IoT Pipeline: MQTT to MongoDB to Dashboard
By Govind Bajaj · 2025

Tags: iot, mqtt, mongodb, architecture, pipeline
I designed an IoT pipeline that ingests 50,000 sensor readings per hour. Three architecture decisions made or broke the system's reliability.
I designed an IoT data pipeline that ingests fifty thousand sensor readings per hour from two hundred industrial sensors.

Three architecture decisions made or broke the system's reliability. Get them right, and the pipeline runs for months without intervention. Get them wrong, and you lose data during the first network hiccup.

This is not a theoretical architecture. It has been running in production for six months.

## Decision 1: MQTT Broker Selection

The MQTT broker is the heart of the pipeline. When it fails, the entire system stops.

Self-hosted Mosquitto is free and lightweight but requires manual scaling and has a single point of failure.

HiveMQ Cloud is managed, has a free tier up to 100 devices, and scales automatically. I chose this for the managed reliability.

AWS IoT Core is enterprise-grade but complex and expensive.

## Decision 2: Message Persistence Strategy

When the backend is down for maintenance, sensor data must not be lost.

I use persistent sessions with cleanSession=false:

```ts
const client = mqtt.connect(brokerUrl, {
  clientId: "backend-processor",
  clean: false,
  qos: 1,
})
```

If the backend restarts, it receives all messages published while it was offline.

## Decision 3: Database Write Strategy

Fifty thousand writes per hour is fourteen writes per second. Individual inserts work but each is a network round-trip.

I use batched inserts:

```ts
class BatchWriter {
  private buffer = []
  private readonly BATCH_SIZE = 100

  add(reading) {
    this.buffer.push(reading)
    if (this.buffer.length >= this.BATCH_SIZE) this.flush()
  }

  private flush() {
    if (this.buffer.length === 0) return
    Reading.insertMany(this.buffer, { ordered: false })
    this.buffer = []
  }
}
```

The buffer flushes at 100 readings or every 5 seconds, whichever comes first.

## The Complete Pipeline Flow

Sensor publishes to MQTT broker. Backend subscribes and receives. Buffer batches readings. MongoDB stores. WebSocket broadcasts. React renders.

Each step is independent and can fail without cascading.

## Takeaways

- Choose a managed MQTT broker unless you have dedicated DevOps resources
- Use persistent sessions (clean: false) to handle backend restarts without data loss
- Batch database writes — individual inserts do not scale
- Design for partial failure — each pipeline stage should operate independently
- Monitor message lag, buffer depth, and sensor heartbeats
- The pipeline is only as reliable as its weakest link
- Three architecture decisions (broker, persistence, writes) determine system reliability