Data Acquisition from an MQTT Broker

Data Scraper

Data Engineer

AWS

AWS Lambda

Python

AMMP Technologies

Data Acquisition From MQTT Broker

Objective

This project entails connecting to remote IoT devices of oil and gas vessels owned by customers of a Netherland based company. We connect to the company's staging server and extract information about the vessels including process variables. Data transformation is also carried out for easier understanding and interpretation. The final output is dumped in JSON format for subsequent analysis.

Conceptual Part

What pros and cons do you see with respect to obtaining readings from an MQTT broker, vs getting them via a REST API?

MQTT is a lightweight publish/subscribe architecture that is designed for resource-constrained devices and low-bandwidth setups. It is used a lot for Internet of Things devices, or other machine-to-machine communication.

Pros:

Designed for resource-constrained devices with small code footprint.

Best for setups with varying level of latency - maximizes available bandwidth.

Fast and efficient message delivery system.

Quick to implement.

Works well for one-to-many simulateneous communcation between devices.

Major cloud service providers support MQTT.

Cons:

Useful for sending only small portions of data to servers.

Authentication is not encrypted by default. Unfortunately, SSL/TLS which isn't lightweight must be employed for security.

It is often difficult to create a globally scalable MQTT network due to its topic structure. As topic tree grows, complexity increases.

MQTT may suffer from high latency projects.

REST APIs can work with files, objects, and media with actions such as POST, GET, PUT, UPDATE, and DELETE; however, MQTT allows for only publishing and subscribing via a broker with smaller data.

How would you run an acquisition function that subscribes to an MQTT broker? For example would you trigger it periodically via a scheduler, or would you have it running as some sort of continuous "listener" function?

The acquisition function in this case is subjective to the existing system. If there are certain periods of the day that an MQTT broker is expected to provide feedback, then, a scheduler is preferred. This solution helps to reduce operational costs by stopping resources that are not in use and starting resources when needed. However, if the MQTT broker is expected to provide feedback at any instance - not a defined period in a day - then a continuous listener function would be preferred.

Practical

Write a script in Python 3, to carry out data acquisition from the MQTT broker, and output the acquired raw data to screen.

Details Programming Language: Python 3

What it Does

The app.py application aims to connect to the MQTT staging server, subscribes to a topic, receives a message dumped to JSON and eventually outputs the necessary information from some remote IoT device about an oil and gas vessel.