** Documentation for MX.
**
** Copyright: (c) 2014-2022 Jacco van Schaik (jacco@jaccovanschaik.net)
**
** This software is distributed under the terms of the MIT license. See
** http://www.opensource.org/licenses/mit-license.php for details.

Introduction

MX is a light-weight communication library written in C. It provides programs that use it with a way to publish and subscribe to messages. There is no predetermined structure to the payload of a message. To MX, a message payload is a simple, opaque byte string.

Design considerations

1. MX is not designed to be in operation 24/7. Although it is possible, with some care, to build such a system using MX, that is not what MX was designed for. It is assumed that systems built on top of MX will be started, run for some time and then be shut down.

   (The reason for this is that message types, once created, are never re-used, which makes them a limited resource. If you design a system that only allocates a fixed number of message types it should be possible to run it indefinitely, but MX will not prevent you from exhausting the available message types. On the other hand, a message type is a 32-bit unsigned integer so it'll take a while to exhaust that supply).

2. MX is not designed to be particularly resilient against external hardware or software failures. It is assumed that a run-time failure in any of the participating components signifies a serious problem that will be rectified before trying again. MX will report the failure but will not try to restart the failed component, for example.

   Also, the master component constitutes an obvious single point of failure. If it crashes or is killed all other participating components will also exit. This should not be a big problem for a standalone master (started using mx master) because it is fairly robust, but if another component assumes the master role (see the mxMaster function) and it exits or crashes, all other participating components will exit too.

3. Finally, security was not a big design consideration for MX. If you can connect to the master component it is trivially easy to kill it with a hand-crafted message (in fact, this message was specifically designed to make it so). This will then cause all other components to exit too. Furthermore, components connect to each other without any authentication, and messages are exchanged without any encryption. MX is expected to run on a private network where all connected devices can be trusted.

TL;DR: If you are planning to use MX for a nuclear power station you should probably reconsider.

Concepts

1. If a valid name (i.e. not a NULL pointer) was given in the call to mxClient or mxMaster, it is used.
2. Otherwise, if the environment variable MX_NAME exists, its contents are used.
3. Otherwise the login name of the current user is used.

The MX name is hashed to generate a port number between 1024 and 65535 inclusive. The master listens on this port on all interfaces and the clients connect to it. It is therefore a fixed "access point" to the message exchange. All the other port numbers used by MX are allocated dynamically by the operating system. This scheme makes it possible to run many message exchanges on a single host (assuming the used MX names don't result in the same hash value, and assuming you don't simply run out of IP ports).

1. If a host name was used in the call to mxClient, it is used.
2. Otherwise, if the environment variable MX_HOST exists, its contents are used.
3. Otherwise "localhost" is used.

A message type is a unique ID assigned to a kind of message (not to individual messages!) They are allocated starting at 0, but message types 0 through 9 are used by MX itself (see the Message structure section for a description of these messages). Message types available to the user therefore start at 10. A message type is a 32-bit unsigned integer, so there are 2³² = 4,294,967,296 message types available in total. If that isn't enough it is also possible to transmit a version number along with the message. The version number is also a 32-bit unsigned integer.

Message types must be registered before they can be used. Components do this by calling the mxRegister function. They can pass in a name by which the message type will henceforth be known. If any other component registers a message with the same name they will get the same message type. This way, message types are coordinated between components.

Components can also register an anonymous message (by using NULL for the message name). This allocates a new message type without associating it with a name. This way components can allocate a message type for their own private use.

Message types are never reused, so it is possible to exhaust the number of message types in a single run, depending of course on how quickly they are used up.

Usage: mx <command> [ <options> ]

Commands:
  help     Show help
  master   Run a master component
  name     Print the effective MX name
  host     Print the effective MX host
  port     Print the effective MX port
  list     Show a list of participating components
  quit     Ask the master component to exit
  version  Show the current version of MX

Use "mx help <command>" to get help on a specific command.

Using MX

In a program that uses MX, there are two phases to be distinguished: first there is some setup, then you enter the message loop.

Setup

In the setup phase you do the following:

Get a pointer to an MX struct by calling mxClient or mxMaster.

Normal components should use mxClient to connect to a master component, giving the hostname where it runs and its MX name, along with the name under which the component itself wants to be known. Components that also want to assume the master role themselves should call mxMaster with just the MX name and their own name. Note, however, that only one component can be the master.

Register the message types you're interested in using mxRegister.

This should be done for the message types that you want to subscribe to and those you want to publish. The mxRegister function returns the message type ID that you should use for all subsequent activities.

Subscribe to the messages you're interested in and possibly to other events.

With every subscription you specify a callback function that should be called whenever a message of the given type is received. You can also subscribe to other events, such as new messages being registered, components connecting to or disconnecting from the master, or components subscribing to or unsubscribing from a message type. With every subscription you can also pass in an opaque void pointer udata (for "user data") which is passed back to you unchanged on every delivery. By the way, there is no additional work for messages you want to send; you don't have to "announce" them or anything. You can simply start sending them.

Message loop

There are two ways to run the message loop: you can call the mxRun function which will handle everything for you and exit when the user calls mxShutdown, or listen for events yourself and call the mxProcessEvents function to have MX handle them.

Using mxRun

This is the simplest method. Simply call the mxRun function after the setup is done, and this will run continuously, calling registered callbacks as necessary, until the mxShutdown function is called or until the connection to the master component is lost.

Using mxProcessEvents

This method can be used if you need to run the application's event loop yourself. You can listen on the file descriptor returned by the mxConnectionNumber function, and when that file descriptor becomes "ready for read" you call the mxProcessEvents function. This will then handle the incoming data and call the appropriate handlers.

By the way, it is perfectly fine to register new messages and/or subscribe to them once the message loop is running.

Function reference

Return the mx_name that would be used if mx_name were to be given to mxClient. If it is a valid name (i.e. not NULL) that same name is returned. Otherwise, if the environment variable MX_NAME is set it is returned. Otherwise, if the environment variable USER (the user's login name) is set it is returned. Otherwise MX gives up and NULL is returned.

Return the listen port that the master component will use for the message exchange with name mx_name. The same algorithm as indicated for mxEffectiveName is used to determine the effective mx_name.

Return the mx_host that would be used if mx_host were to be given to mxClient or mxMaster. If it is a valid name (i.e. not NULL) it is returned. Otherwise, if the environment variable MX_HOST is set it is returned. Otherwise "localhost" is returned.

This "backgrounding" is done by using a fork() system call, which has the unfortunate consequence that any threads started before the call to mxMaster will be killed. So if you want to background the master component and also start additional threads, do the latter after calling this function.

Return the name of the local component, as passed to mxClient or mxMaster as my_name.

Create a timer that will call handler at time t (seconds since the UNIX epoch). In future calls to mxAdjustTimer and mxRemoveTimer this timer will be identified by id. When calling handler, the same pointer udata given here will be passed back.

Shut down mx. After this function is called, the mxRun function will return.

Destroy mx. Call this function only after mxRun has returned.

Internals

Message structure

MX messages consist of two parts: the header and the payload. The header has a fixed size of 12 bytes and looks like this:

It contains:

• Message type (big-endian unsigned 32-bit integer)
• Message version (big-endian unsigned 32-bit integer)
• Payload length (big-endian unsigned 32-bit integer)

The header is followed by the payload, consisting of as many bytes as was given in the payload size. MX imposes no structure whatsoever on the payload. To MX, the payload is an opaque string of bytes.

All messages are exchanged over TCP/IP connections.

Requests

Requests are sent by normal components to the master.

Replies

Replies are sent by the master to normal components as a reply to a request.

Reports

Reports are sent by the master to normal components to inform them of a change in the message exchange.

Updates

Updates are sent between components (including the master) to inform each other of changes in their configuration.

All integers in these messages are sent in big-endian or network byte order format, with the most significant byte transmitted first.

QuitRequest (type 0)

This message type asks the master component to quit. It has type 0, version 0 and no payload, so it is the simplest message type, consisting of just 12 null-bytes. If the master component quits, all attached components will also quit when they see their network connection to the master go down.

HelloRequest (type 1)

This message is sent by normal components to the master to introduce themselves. It contains the component's name and the port on which it can be reached (the master determines the new component's IP address by looking at the connection over which this message is received).

HelloReply (type 2)

This message is sent by the master as a reply to a normal components HelloRequest message. It informs the component of the name that the master is running under. For the standalone master, started using mx master, this is always "master". But if another component has assumed the role of master (by calling mxMaster instead of mxClient during setup), this may be a different name.

HelloReport (type 3)

This message is sent by the master to all connected components to report a new component.

HelloUpdate (type 4)

This message is sent by clients after they've connected to another client to introduce themselves.

RegisterRequest (type 5)

This message is sent by normal components to the master to register a message type.

RegisterReply (type 6)

This message is sent by the master to a normal component as a reply to a register request.

RegisterReport (type 7)

This message is sent from the master to a normal component to report a newly registered message.

SubscribeUpdate (type 8)

This message is sent between normal components to tell the recipient about a new subscription by the sender.

CancelUpdate (type 9)

This message is sent between normal components to tell the recipient about a cancelled subscription by the sender.

Registering messages

Messages are identified with a name and a numeric message type. Registering a message is a way to link an ASCII name (which is more convenient for the user) with an integer type (which is more convenient for the system). Each name has one and only one associated type, so if multiple components register messages with the same name, they are guaranteed to get the same type. Messages are registered once before their first use, and solely referenced by their type after that.

Messages are registered using the mxRegister function. First of all, this function checks the local database to see if it already knows about the given message, If so, it immediately returns the associated message type to the caller. Otherwise it sends a RegisterRequest message to the master.

When the master receives the RegisterRequest message, it checks its database for the given message name. If found, it retrieves the associated type. Otherwise a new type is generated (simply by increasing a counter) and stored in the database, along with the name. The message type is then returned to the component in a RegisterReply message.

If the master had to generate a new type it also sends all other existing components a RegisterReport message to inform them of the new message type (this is the reason that a component might already have the message type in its local database, even though no mxRegister call has yet been made for it).

When the component receives the RegisterReply message it stores the message's name and associated type in its database, and the mxRegister function returns the new message type.

All message names are associated with a unique message type, but not all message types have to have an associated message name. Components can ask the master for a unique message type that isn't associated with a message name. They do this by calling the mxRegister function with a NULL message name, and they will get a unique message type every time. This can then be distributed to other components, and used without the fear of getting message type collisions.

Initial connection

This section describes how a new component is added to an existing message exchange. We'll assume that we already have a master (M) and a component (C1) running. These are the events that take place when the new component, C2 calls the mxClient function:

1. mxClient opens a listen socket, but without binding it to a port number, in which case the operating system chooses a random free port and binds the listen socket to it. The mxClient function then connects to the master and sends a HelloRequest message containing the component's name and the listen port number that the OS has chosen. The master stores these attributes for C2, along with its host name, which it determines by looking at the "peer name" of the component's TCP connection.

2. The master replies to the HelloRequest message with a HelloReply message. This contains the name that the master is running under.

   Usually, the master role will be fullfilled by the mx executable, and in this case its name will simply be "master". But it is also possible for another component to fullfill this role (by calling the mxMaster function instead of mxClient), and in that case it may be running under a different name.

3. For each component that the master already knows about (only C1 in this example but there can be more), it sends a HelloReport back to C2. These messages contain the name, hostname and listen port of an existing component.

4. Next, the master sends a RegisterReport message to C2 for each message type that has already been registered.

5. Finally, the master sends C2 a SubscribeUpdate message for each of its subscriptions.

At this point the master component considers its work done and returns to listening for new connections and messages.

6. For each HelloReport message that was received, the mxClient function connects to the component described in the message and sends it a HelloUpdate message to introduce the new component C2. This message contains only C2's name.

7. It then sends the existing component a SubscribeUpdate message for each of its subscriptions.

8. At the same time, C1 responds to the new connection by sending its subscriptions to C2, again using SubscribeUpdate messages

After this, C2 is fully integrated into the MX system and mxClient returns.

It is worth noting that the SubscribeUpdate messages being sent to-and-fro are not unique to the setup phase. If a component calls the mxSubscribe function at any later time, the same message type is used to communicate the new subscription to all other components.

Threads

This section describes the threads associated with an MX component.

First of all, the MX main loop runs in the application's main thread. It may consist of the application repeatedly calling mxProcessEvents whenever data is available on the file descriptor returned by mxConnectionNumber, or of the application's single call to the mxRun function, which does essentially the same.

In addition, a number of separate threads handle communication with the outside world. The main loop sends commands (if necessary) to these threads using command queues, and they report back events through an event pipe (a standard UNIX pipe). This means that any event that the main loop needs to respond to comes in through a single file descriptor, and it is this file descriptor that is returned by the mxConnectionNumber function. When an event does come in, a subsequent call to the mxProcessEvents function reads it from the event pipe and handles it.

The following threads exist:

A timer thread.

This thread sets up and waits for timers, as instructed by the main loop. When a timer times out it sends an event back to the main loop informing it of this.

A listener thread.

This thread's sole responsibility is to listen on the message exchanges listen socket, and inform the main loop of new connection requests.

Then, for every connection to another component there are two additional threads:

A reader thread.

This thread receives incoming messages from the connected component and sends them on to the main loop.

A writer thread.

This thread sends messages out to the connected component, as instructed by the main loop.

The timer and writer threads exit when an explicit "exit" command comes in over their command queue. The listener and reader threads exit when the main loop shuts down the TCP sockets that they are connected to.