Common message types and definitions for federated Lingua Franca programs. More...

Macros
#define	ADDRESS_QUERY_RETRY_INTERVAL MSEC(250)
	Time that a federate waits before asking the RTI again for the port and IP address of a federate.
#define	DELAY_START SEC(1)
	Delay the start of all federates by this amount.
#define	ENCODE_STOP_GRANTED(buffer, time, microstep)
	Encode a stop granted message.
#define	ENCODE_STOP_REQUEST(buffer, time, microstep)
	Encode a stop request message.
#define	ENCODE_STOP_REQUEST_REPLY(buffer, time, microstep)
	Encode a stop request reply message.
#define	FED_COM_BUFFER_SIZE 256u
	Size of the buffer used for messages sent between federates.
#define	FEDERATE_ID_IN_USE 2
	Code sent with a MSG_TYPE_REJECT message indicating that the federate ID is already in use.
#define	FEDERATE_ID_OUT_OF_RANGE 3
	Code sent with a MSG_TYPE_REJECT message indicating that the federate ID is out of range.
#define	FEDERATION_ID_DOES_NOT_MATCH 1
	Code sent with a MSG_TYPE_REJECT message indicating that the federation ID does not match.
#define	HMAC_DOES_NOT_MATCH 6
	Code sent with a MSG_TYPE_REJECT message indicating that the HMAC authentication failed.
#define	MSG_TYPE_ACK 255
	Byte identifying an acknowledgment of the previously received message.
#define	MSG_TYPE_ADDRESS_ADVERTISEMENT 15
	Byte identifying a message advertising the port for the TCP connection server of a federate.
#define	MSG_TYPE_ADDRESS_QUERY 13
	Byte identifying a address query message, sent by a federate to RTI to ask for another federate's address and port number.
#define	MSG_TYPE_ADDRESS_QUERY_REPLY 14
	Byte identifying a address query message reply, sent by a RTI to a federate to reply with a remote federate's address and port number.
#define	MSG_TYPE_CLOCK_SYNC_CODED_PROBE 22
	Coded probe message.
#define	MSG_TYPE_CLOCK_SYNC_T1 19
	Physical clock synchronization messages according to PTP.
#define	MSG_TYPE_CLOCK_SYNC_T3 20
	Prompt the master to send a T4.
#define	MSG_TYPE_CLOCK_SYNC_T4 21
	Physical clock synchronization message according to PTP.
#define	MSG_TYPE_DOWNSTREAM_NEXT_EVENT_TAG 26
	Byte identifying a downstream next event tag (DNET) message sent from the RTI in centralized coordination.
#define	MSG_TYPE_FAILED 25
	Byte identifying that the federate or the RTI has failed.
#define	MSG_TYPE_FED_IDS 1
	Byte identifying a message from a federate to an RTI containing the federation ID and the federate ID.
#define	MSG_TYPE_FED_NONCE 100
	Byte identifying a message from a federate to an RTI containing federate's 8-byte random nonce for HMAC-based authentication.
#define	MSG_TYPE_FED_RESPONSE 102
	Byte identifying a message from federate to RTI as a response to the RTI_RESPONSE message.
#define	MSG_TYPE_LATEST_TAG_CONFIRMED 9
	Byte identifying a latest tag confirmed (LTC) message sent by a federate to the RTI.
#define	MSG_TYPE_MESSAGE 3
	Byte identifying a message to forward to another federate.
#define	MSG_TYPE_NEIGHBOR_STRUCTURE 24
	A message that informs the RTI about connections between this federate and other federates where messages are routed through the RTI.
#define	MSG_TYPE_NEIGHBOR_STRUCTURE_HEADER_SIZE 9
	The size of the header of a neighbor structure message.
#define	MSG_TYPE_NEXT_EVENT_TAG 6
	Byte identifying a next event tag (NET) message sent from a federate in centralized coordination.
#define	MSG_TYPE_P2P_MESSAGE 17
	Byte identifying a message to send directly to another federate.
#define	MSG_TYPE_P2P_SENDING_FED_ID 16
	Byte identifying a first message that is sent by a federate directly to another federate after establishing a socket connection to send messages directly to the federate.
#define	MSG_TYPE_P2P_TAGGED_MESSAGE 18
	Byte identifying a timestamped message to send directly to another federate.
#define	MSG_TYPE_PORT_ABSENT 23
	A port absent message, informing the receiver that a given port will not have event for the current logical time.
#define	MSG_TYPE_PROVISIONAL_TAG_ADVANCE_GRANT 8
	Byte identifying a provisional time advance grant (PTAG) sent by the RTI to a federate in centralized coordination.
#define	MSG_TYPE_REJECT 0
	Byte identifying a rejection of the previously received message.
#define	MSG_TYPE_RESIGN 4
	Byte identifying that the federate or the RTI is ending its execution.
#define	MSG_TYPE_RTI_RESPONSE 101
	Byte identifying a message from RTI to federate as a response to the FED_NONCE message.
#define	MSG_TYPE_STOP_GRANTED 12
	Byte sent by the RTI indicating that the stop request from some federate has been granted.
#define	MSG_TYPE_STOP_GRANTED_LENGTH (1 + sizeof(instant_t) + sizeof(microstep_t))
	The length of a stop granted message.
#define	MSG_TYPE_STOP_REQUEST 10
	Byte identifying a stop request.
#define	MSG_TYPE_STOP_REQUEST_LENGTH (1 + sizeof(instant_t) + sizeof(microstep_t))
	The length of a stop request message.
#define	MSG_TYPE_STOP_REQUEST_REPLY 11
	Byte indicating a federate's reply to a MSG_TYPE_STOP_REQUEST that was sent by the RTI.
#define	MSG_TYPE_STOP_REQUEST_REPLY_LENGTH (1 + sizeof(instant_t) + sizeof(microstep_t))
	The length of a stop request reply message.
#define	MSG_TYPE_TAG_ADVANCE_GRANT 7
	Byte identifying a time advance grant (TAG) sent by the RTI to a federate in centralized coordination.
#define	MSG_TYPE_TAGGED_MESSAGE 5
	Byte identifying a timestamped message to forward to another federate.
#define	MSG_TYPE_TIMESTAMP 2
	Byte identifying a timestamp message, which is 64 bits long.
#define	MSG_TYPE_TIMESTAMP_LENGTH (1 + sizeof(int64_t))
	The length of a timestamp message.
#define	MSG_TYPE_UDP_PORT 254
	Byte identifying an acknowledgment of the previously received MSG_TYPE_FED_IDS message.
#define	NONCE_LENGTH 8
	The randomly created nonce size will be 8 bytes.
#define	RTI_NOT_EXECUTED_WITH_AUTH 7
	Code sent with a MSG_TYPE_REJECT message indicating that the RTI was not executed using the -a or –auth option.
#define	SHA256_HMAC_LENGTH 32
	The HMAC tag uses the SHA256 hash algorithm, creating a 32 byte length hash tag.
#define	UNEXPECTED_MESSAGE 4
	Code sent with a MSG_TYPE_REJECT message indicating that the incoming message is not expected.
#define	WRONG_SERVER 5
	Code sent with a MSG_TYPE_REJECT message indicating that the connected to the wrong server.

Detailed Description

Common message types and definitions for federated Lingua Franca programs.

Author: Edward A. Lee; Soroush Bateni

This file defines the message types for the federate to communicate with the RTI. Each message type has a unique one-byte ID.

Startup sequence

The startup sequence is as follows:

Establishing a TCP connection

Each federate attempts to connect with an RTI at the IP address put into its code by the code generator (i.e., it attempts to open a TCP connection). If an explicit port is given in the at clause on the federated reactor statement, it will use that port. Otherwise, it will use DEFAULT_PORT.

When it has successfully opened a TCP connection, the first message it sends to the RTI is a MSG_TYPE_FED_IDS message, which contains the ID of this federate within the federation, contained in the global variable _lf_my_fed_id in the federate code (which is initialized by the code generator) and the unique ID of the federation, a GUID that is created at run time by the generated script that launches the federation. If you launch the federates and the RTI manually, rather than using the script, then the federation ID is a string that is optionally given to the federate on the command line when it is launched. The federate will connect successfully only to an RTI that is given the same federation ID on its command line. If no ID is given on the command line, then the default ID "Unidentified Federation" will be used.

The RTI will respond with a MSG_TYPE_REJECT message if the federation IDs do not match and close the connection. At this point the federate will increment the port number and try again to find an RTI that matches.

When the federation IDs match, the RTI will respond with an MSG_TYPE_ACK.

Conveying the neighbor structure

The next message to the RTI will be a MSG_TYPE_NEIGHBOR_STRUCTURE message that informs the RTI about connections between this federate and other federates where messages are routed through the RTI. Currently, this only includes logical connections when the coordination is centralized. This information is needed for the RTI to perform the centralized coordination. The burden is on the federates to inform the RTI about relevant connections.

The next message to the RTI will be a MSG_TYPE_UDP_PORT message, which has payload USHRT_MAX if clock synchronization is disabled altogether, 0 if only initial clock synchronization is enabled, and a port number for UDP communication if runtime clock synchronization is enabled. By default, if the federate host is identical to that of the RTI (either no "at" clause is given for either or they both have exactly the same string), then clock synchronization is disabled. Otherwise, the default is that initial clock synchronization is enabled. To turn turn off clock synchronization altogether, set the clock-sync property of the target to off. To turn on runtime clock synchronization, set it to on. The default value is initial.

Clock synchronization

If initial clock sync is enabled, the next step is to perform the initial clock synchronization (using the TCP connection), which attempts to find an initial offset to the physical clock of the federate to make it better match the physical clock at the RTI.

Clock synchronization is initiated by the RTI by sending a message of type MSG_TYPE_CLOCK_SYNC_T1, the payload of which is the current physical clock reading at the RTI. The federate records the physical time when it receives this message (T2) and sends a reply message of type MSG_TYPE_CLOCK_SYNC_T3 to the RTI. It records the time (T3) at which this message has gone out. The payload of the MSG_TYPE_CLOCK_SYNC_T3 message is the federate ID. The RTI responds to the T3 message with a message of type MSG_TYPE_CLOCK_SYNC_T4, which has as a payload the physical time at which that response was sent. This cycle will happen _LF_CLOCK_SYNC_EXCHANGES_PER_INTERVAL times at startup to account for network delay variations (see below).

The times T1 and T4 are taken from the physical clock at the RTI, whereas the times T2 and T3 are taken from the physical clock at the federate. The round trip latency on the connection to the RTI is therefore measured as (T4 - T1) - (T3 - T2). Half this quantity is an estimate L of the one-way latency. The estimated clock error E is therefore L - (T2 - T1). Over several cycles, the average value of E becomes the initial offset for the clock at the federate. Henceforth, when lf_time_physical() is called, the offset will be added to whatever the physical clock says.

If clock synchronization is enabled, then the federate will also start a thread to listen for incoming UDP messages from the RTI. With period given by the -c on period <n> command-line argument, the RTI will initiate a clock synchronization round by sending to the federate a MSG_TYPE_CLOCK_SYNC_T1 message. A similar protocol to that above is followed to estimate the average clock synchronization error E, with two exceptions. First, a fraction of E (given by _LF_CLOCK_SYNC_ATTENUATION) is used to adjust the offset up or down rather than just setting the offset equal to E. Second, after MSG_TYPE_CLOCK_SYNC_T4, the RTI immediately sends a following message of type MSG_TYPE_CLOCK_SYNC_CODED_PROBE. The federate measures the time difference between its receipt of T4 and this code probe and compares that time difference against the time difference at the RTI (the difference between the two payloads). If that difference is larger than CLOCK_SYNC_GUARD_BAND in magnitude, then the clock synchronization round is skipped and no adjustment is made. The round will also be skipped if any of the expected UDP messages fails to arrive.

See: Geng, Y., et al. (2018). Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization. USENIX Symposium on Networked Systems Design and Implementation (NSDI), Renton, WA, USA.

Setting up coordination

The next step depends on the coordination mode. If the coordination parameter of the target is "decentralized" and the federate has inbound connections from other federates, then it starts a socket server to listen for incoming connections from those federates. It then sends to the RTI an MSG_TYPE_ADDRESS_ADVERTISEMENT message with the port number as a payload. The federate then creates a thread to listen for incoming socket connections and messages.

If the federate has outbound connections to other federates, then it establishes a socket connection to those federates. It does this by first sending to the RTI an MSG_TYPE_ADDRESS_QUERY message with the payload being the ID of the federate it wishes to connect to. If the RTI responds with a -1, then the RTI does not (yet) know the remote federate's port number and IP address, so the local federate will try again after waiting ADDRESS_QUERY_RETRY_INTERVAL. When it gets a valid port number and IP address in reply, it will establish a socket connection to that remote federate.

Physical connections also use the above P2P sockets between federates even if the coordination is centralized.

Afterward, the federates and the RTI decide on a common start time by having each federate report a reading of its physical clock to the RTI on a MSG_TYPE_TIMESTAMP. The RTI broadcasts the maximum of these readings plus DELAY_START to all federates as the start time, again on a MSG_TYPE_TIMESTAMP.

The next step depends on the coordination type.

Under centralized coordination, each federate will send a MSG_TYPE_NEXT_EVENT_TAG to the RTI with the start tag. That is to say that each federate has a valid event at the start tag (start time, 0) and it will inform the RTI of this event. Subsequently, at the conclusion of each tag, each federate will send a MSG_TYPE_LATEST_TAG_CONFIRMED followed by a MSG_TYPE_NEXT_EVENT_TAG (see the comment for each message for further explanation). Each federate would have to wait for a MSG_TYPE_TAG_ADVANCE_GRANT or a MSG_TYPE_PROVISIONAL_TAG_ADVANCE_GRANT before it can advance to a particular tag.

Under decentralized coordination, the coordination is governed by STA and STAAs, as further explained in https://doi.org/10.48550/arXiv.2109.07771.

FIXME: Expand this. Explain port absent reactions.

Requesting a stop

Overview of the algorithm: When any federate calls lf_request_stop(), it will send a MSG_TYPE_STOP_REQUEST message to the RTI, which will then forward a MSG_TYPE_STOP_REQUEST message to any federate that has not yet provided a stop time to the RTI. The federates will reply with a MSG_TYPE_STOP_REQUEST_REPLY and a stop tag (which shall be the maximum of their current logical tag at the time they receive the MSG_TYPE_STOP_REQUEST and the tag of the stop request). When the RTI has gathered all the stop tags from federates (that are still connected), it will decide on a common stop tag which is the maximum of the seen stop tag and answer with a MSG_TYPE_STOP_GRANTED. The federate sending the MSG_TYPE_STOP_REQUEST and federates sending the MSG_TYPE_STOP_REQUEST_REPLY will freeze the advancement of tag until they receive the MSG_TYPE_STOP_GRANTED message, in which case they might continue their execution until the stop tag has been reached.

Macro Definition Documentation

◆ MSG_TYPE_STOP_REQUEST

#define MSG_TYPE_STOP_REQUEST 10

Byte identifying a stop request.

This message is first sent to the RTI by a federate that would like to stop execution at the specified tag. The RTI will forward the MSG_TYPE_STOP_REQUEST to all other federates. Those federates will either agree to the requested tag or propose a larger tag. The RTI will collect all proposed tags and broadcast the largest of those to all federates. All federates will then be expected to stop at the granted tag.

The next 8 bytes will be the timestamp. The next 4 bytes will be the microstep.

NOTE: The RTI may reply with a larger tag than the one specified in this message. It has to be that way because if any federate can send a MSG_TYPE_STOP_REQUEST message that specifies the stop time on all other federates, then every federate depends on every other federate and time cannot be advanced. Hence, the actual stop time may be nondeterministic.

If, on the other hand, the federate requesting the stop is upstream of every other federate, then it should be possible to respect its requested stop tag.

◆ MSG_TYPE_STOP_REQUEST_REPLY

#define MSG_TYPE_STOP_REQUEST_REPLY 11

Byte indicating a federate's reply to a MSG_TYPE_STOP_REQUEST that was sent by the RTI.

The payload is a proposed stop tag that is at least as large as the one sent to the federate in a MSG_TYPE_STOP_REQUEST message.