ECE 448/528 Application Software Design Lecture 11 The Publish-Subscribe Pattern and MQTT (MQ Telemetry Transport)

(1)

ECE 448/528 – Application Software Design

Lecture 11 The Publish-Subscribe Pattern

and MQTT (MQ Telemetry Transport)

Professor Jia Wang

Department of Electrical and Computer Engineering

Illinois Institute of Technology

(2)

Outline

The Publish-Subscribe Pattern

(3)

Outline

The Publish-Subscribe Pattern

(4)

The Publish-Subscribe Pattern

I

A design pattern to further decompose event processing and

event delivery.

I Usually on top of network communications.

I

Publisher: parties that produce events

I

Subscriber: parties that consume events

I

Message broker: a service that collects events from publishers

and delivers them to subscribers.

I A.k.a. event bus, message queue middleware, etc.

I Events are usually encoded into string/byte messages so that a message broker may work independent of specific applications.

(5)

Observer Pattern Revisited

I

In comparison with the observer pattern

I EventSource is similar to publisher as both produce events. I Observer is similar to subscriber as both consume events.

I

However, unlike the observer pattern where the EventSource

knows and manages the observers, publishers and subscribers

are fully decoupled.

I

Publishers only communicate with the message broker to send

events.

I Publishers have no idea what subscribers are consuming the events they produce.

I

Subscribers only communicate with the message broker to

receive events.

I Subscribers have no interests on who produce the events they consume.

I

How does the message broker know to which subscribers those

events should be delivered to?

(6)

Topic Management

I

For each event, in addition to the message representing the

event, the publisher need to specify a topic.

I The topic usually is a meaningful string providing hints on what this message/event is all about.

I For each event, the publisher sends (topic, message) to the message broker.

I

Each subscriber, when establishing connection with the

message broker, should specify what topics it is interested

into.

I Then the message broker will only send (topic, message) with matching topics to this subscriber.

(7)

Topic Management

I

For each event, in addition to the message representing the

event, the publisher need to specify a topic.

I The topic usually is a meaningful string providing hints on what this message/event is all about.

I For each event, the publisher sends (topic, message) to the message broker.

I

Each subscriber, when establishing connection with the

message broker, should specify what topics it is interested

into.

I Then the message broker will only send (topic, message) with matching topics to this subscriber.

(8)

Topic Routing

I

A subscriber may simple specify all topics it is interested into.

I

This simply strategy may fail:

I When there are too many such topics.

I When new publishers start to publish to new topics that the subscriber may be interested into.

I

A message broker may allow subscribers to subscribe to a

wildcard of topics.

I Whether the subscriber has the capacity to process messages from all matching topics is not of concern here.

(9)

Delivery Guarantee

I

Should message broker acknowledge publisher and should

subscriber acknowledge message broker of message being

received?

I A fundamental problem of network communication and messaging systems.

I

Delivery guarantee

I At least once: resend messages not acknowledged I Exactly once: how?

I At most once: don’t acknowledge at all

I

It is up to the applications to choose a proper delivery

guarantee.

I And we may need to redesign an application if the delivery guarantee cannot be achieved.

(10)

Delivery Guarantee: Exactly Once

I

The publisher may include a sequence number for each

message.

I

Publisher/message broker/subscribers all utilize at least once

delivery.

I

Message broker and subscribers use the sequence number to

remove duplicated messages to achieve exactly once delivery.

(11)

Message Ordering

I

Assume network communications always arrive in order.

I But some messages may get lost and need to be resent

according to delivery guarantee.

I

At most once: messages always arrive in order

I

Exactly once: messages may arrive out of order

I Order can be recovered via the sequence number.

I

At least once: depend on performance requirement

I Wait for acknowledgement before sending a new message:

in-order arrival but low performance.

I Allow multiple unacknowledged messages: better performance but a lost message being resent may arrive out of order.

(12)

Persistence

I

A subscriber may need to receive all historical messages

published to a topic.

I As needed by the application.

I

A message broker may support persistence, storing all

historical messages.

I Persistence may require substantial amount of storage.

I

It may take a lot of time for a subscriber to receive all

historical messages.

I What if the subscriber fails and need to restart?

I Subscriber local state: assume messages are with sequence numbers, then a subscriber may store messages locally and request the message broker to send messages after the last one upon restarting.

(13)

Retained Messages

I

Without persistence, a subscriber may need to wait

indefinitely before a message is published.

I Could be a problem if the subscriber need to decide the status of the publisher and the publisher cannot publish its status periodically.

I

Retained messages: the message broker only persists the last

message on a topic.

I A lightweight alternative to persistence for some cases. I Either as requested by the publisher or as configured. I New subscribers will receive it as the first message.

(14)

Multiple Publishers and Multiple Subscribers

I

Multiple publishers are usually allowed to publish to the same

topic.

I There should be no communcation between publishers. I No guarantee on ordering of messages from different

publishers.

I Each publisher should maintain its own sequence number if needed.

I

Multiple subscribers subscribe to the same topic are not aware

of each other.

(15)

Multiple Message Brokers

I

It is possible to utilize multiple message brokers.

I Improve system performance when there are more publishers and subscribers.

I Provide failover if a message broker fails.

I

Publishers and subscribers need to be aware of those message

brokers.

I Usually they would utilize timeout to decide if a message broker is failing and to switch to a different one if so.

I

The message brokers will need to rely on certain consensus

protocols to synchronize received messages.

(16)

Outline

The Publish-Subscribe Pattern

(17)

Message Broker Design Trade-Offs

I

With so many features to choose from and many of which

affecting each other, there are many message broker products

making different design trade-offs.

I

Usually you will find two extremes.

I A lightweight one that focuses on simplifying message delivery but provides no persistence or failover.

I A full-featured one that would meet all needs by providing persistence and failover.

I

Which one is better? It all depends on what is required by

your application.

(18)

MQTT (MQ Telemetry Transport)

I

A lightweight publish-subscribe protocol.

I Designed in late ’90s for devices with limited resource and bandwidth.

I An open OASIS and ISO standard and widely used for IoT devices now.

I

Architecture

I A MQTT broker to relay the messages.

I Clients connected to the MQTT broker to publish messages and subscribe to topics.

(19)

MQTT Features

I

Topic: a UTF-8 string

I Consist of topic levels separated by /. I E.g. iot_ece448/action/plugA/on

I Clients may use wildcard to subsribe for multiple topics, e.g. iot_ece448/action/# for any topics starting with

iot_ece448/action/

I

Support at most once, at least once, and exactly once

delivery.

I But please keep in mind since persistence is not supported, all delivery guarantees do not apply to past messages, in