Summary
- A proof of concept for loss-free sensor data ingestion in a smart home architecture, made for the TU Delft Software Architecture course. I built most of the proof of concept and wrote the report sections on it.
- Sensors use store-and-forward with a persistent local buffer and chunked resynchronisation, sending readings over HTTP to a FastAPI service that validates them with Pydantic and publishes them to a durable RabbitMQ queue read by acknowledging consumers.
- Fault-injection scenarios, unit tests with mocked HTTP failures and load tests (5,000 readings in one request, 200 concurrent requests, 1,000 messages from 50 sensors) confirm that every disconnected sensor empties its buffer and every request is accepted.
- A three-stage GitLab CI pipeline runs style checks, builds the Docker image and runs all tests inside it against a RabbitMQ service.
Context
For the Software Architecture course our team designed a smart home system that learns household routines from sensor data. To keep sensitive data under the user's control while still allowing remote access, the design splits the system into a local hub, which handles data collection, storage, routine learning and device communication, and a set of cloud services for remote access, authentication, updates and notifications. The hub follows a microkernel architecture, with a core and independent plug-ins, and the cloud side a microservice architecture. I built most of the proof of concept and wrote the report sections on it.
The proof of concept tests one component of the hub's network module: delivering sensor readings to the backend without loss when the home's network connection drops. It targets the design's availability and scalability goals, and two problems in particular:
- Data loss during outages. A sensor that sends directly to a server loses its readings when the connection drops.
- Tight coupling. If the service that receives readings also processes them, a slow or failed processor stalls ingestion.
Availability is defined as sensors continuing to produce readings while disconnected, with every reading reaching the queue once the connection returns.
Architecture
sensor (buffer) -> HTTP batch -> ingestion service (validate) -> durable queue -> consumer (ack)
- Ingestion service. A stateless FastAPI REST endpoint that accepts a batch of readings, validates each against a Pydantic schema (reading identifier, sensor identifier, type, value and timestamp) and publishes it to RabbitMQ as a persistent message. It returns 503 when the broker is unreachable, so the sensor treats a broker outage like a network outage, and exposes a health endpoint.
- Message queue. A durable RabbitMQ queue with persistent messages decouples ingestion from processing, absorbs bursts and survives broker restarts.
- Consumer. Takes one message at a time (a prefetch of one, for fair dispatch) and acknowledges it only after processing, reconnecting automatically when the broker is lost.
- Sensors. Simulated sensors generate a reading every one to two seconds and implement store-and-forward with a persistent local buffer.
The design choices follow from the two problems. Keeping the ingestion service's role minimal (accept, validate, forward) lets it handle many concurrent connections without being slowed by processing, and validation at entry acts as a gateway, so malformed data never enters the queue. The queue lets ingestion and processing scale independently, and since all consumers read from the same queue as competing consumers, processing capacity scales horizontally by adding instances. Manual acknowledgement gives at-least-once delivery between queue and consumer: if a consumer fails before acknowledging, the broker requeues the message.
The broker, ingestion service and consumer run as separate containers under Docker Compose, and the services start only once the broker passes its health check.
Transport choice
The report compares four transports between sensors and the ingestion service, judged on reliability under intermittent connectivity, observability and operational complexity:
| Protocol | For | Against |
|---|---|---|
| REST | Firewall friendly, simple request and response with explicit chunked backlog upload, easy validation and backoff | More overhead per message than MQTT, not a push protocol |
| MQTT | Designed for IoT, persistent sessions, low bandwidth and battery use | Needs a broker, more moving parts, ordering subtleties at scale |
| WebSockets | Low latency, server push, full duplex | Needs backpressure and reconnection logic, harder to batch |
| UDP | Lightweight, suits constrained networks | Network traversal, harder to observe |
REST was chosen for its ease of handling and its suitability for a smart home application.
Buffering and resynchronisation
Store-and-forward
Each sensor keeps its backlog as an in-memory queue mirrored to an append-only log file, so the buffer also survives a restart of the sensor and is reloaded on start. When a send fails, through a connection error, a timeout or a server error such as a 503, the sensor marks itself offline and appends the reading to its buffer. While offline, every new reading is buffered rather than sent.
Chunked resynchronisation
Once the connection is back, the sensor drains the backlog oldest first in chunks of 25 readings. On each cycle it sends one chunk, removes those items and rewrites the buffer file only after the server has accepted them, and then sends the current reading, so the backlog drains alongside live data rather than delaying it. A failed chunk leaves the buffer untouched and in order, the sensor returns to offline mode, and the new reading is appended to the end, so nothing is lost or reordered.
Chunking keeps a sensor that has been offline for a long time from overloading the server, spreads the load when many sensors reconnect at once, and limits a failure partway through to retrying the remaining chunks rather than the whole backlog. Every reading carries its own UUID, assigned by the sensor when it is generated.
Evaluation
The evaluation tests the store-and-forward logic in isolation, the whole pipeline under failure, and the whole pipeline under load.
Fault injection
An automated scenario starts the server, consumer and five sensors and runs three phases: 10 seconds of normal operation as a baseline, a 20 second disruption in which three sensors go offline while the others keep sending, and 20 seconds of resynchronisation in which all three upload their backlogs at once. It passes when every disconnected sensor has emptied its buffer. A second scenario disconnects and reconnects two sensors at overlapping times.
Unit tests
Mocked HTTP responses test the sensor without a live server: the buffer is reloaded on start, connection errors and 503 responses both lead to buffering and offline mode, a successful chunk removes exactly the readings sent, and a failed chunk stays in the buffer with new readings appended after it.
Load tests
Run against the full architecture of server, broker and consumer, the load tests cover three scenarios. The first sends 5,000 readings in one request, as from a sensor resynchronising a large backlog. The second sends 200 concurrent requests, as from many sensors sending at once. The third is a sustained stream of 1,000 messages from 50 sensors, which checks that the consumer keeps pace with publishing so the queue does not grow without bound. Each passes only if every request is accepted.
Continuous integration
A three-stage GitLab CI pipeline runs a style stage (compilation check and black), a build stage that builds the Docker image and pushes it to the GitLab container registry, and a test stage that runs the unit, load and fault tests inside that image against a RabbitMQ service.
Scope
The sensors are simulated and the consumer only acknowledges messages. Next steps in the report are a consumer that forwards readings for analysis and storage, and device authentication at the ingestion service so that only authorised devices can send data.