Overview

The Session Store handles handles Ethernet frames and tcp sessions. Captured raw Ethernet frames are categorized, and filtered by various blacklists and whitelists. The set of frames that belong to a single TCP instance are binned together and sorted. Several other stages of processing later, the TCP payload is presented as a range of bytes for output.

Sockets

Reflector binds promiscuously to a pair of Ethernet sockets and traffics frames between them. It requires exclusive access to both sockets. If any other process (including another Reflector) binds to it, a packet storm can result. As a result, Reflector attempts to negotiate unique access via flock() for the entire duration.

Protocol Filters

Reflector inspect packets from the outside in, and iteratively up the OSI layers. At each level, there are three possible resolutions:

• Rejected - packet swallowed
• Inspected - packet contents passed onto the next stage of inspection
• Ignored - packet forwarded without any further inspection

TCP Whitelists & Network Address Translation

The two sockets bound by Reflector are not fungible. One of them leads to traders (ingress), the other leads to venues (egress). Packets transiting from trader to venue are "OUTBOUND". Packets transiting from venue to trader are "INBOUND". Both ends are independently whitelisted, and the addresses on both ends are independently translated (with exceptions).

On the venue side, a whitelist on ordered pair of ( IPv4Addr, Port ) is enforced, and every pair is translated (possibly to itself). The set of allowed original venue ip/port pairs must not emit repeats and the set of allowed translated venue IP/port pairs must not emit repeats. However, the two sets reside in completely different namespaces.

On the trader side, a whitelist on IP address is enforced, and every address is translated (possibly to itself). The set of allowed original trader IP addresses must be unique while the set of allowed translated trader IP addresses can overlap at will. This is because natted trader-side ports are assigned by Reflector.

Original trader ports are unrestricted, but translated trader ports are assigned by Reflector. This is possible because the all whiltelisted TCP streams (except for BGP sessions) must have the trader as client and the venue as server. The initial SYN packet is made from an original trader address pair to a natted venue address pair. An unique natted port is generated for that session, and the headers
of that packet gets rewritten to contain the natted trader address pair and the original venue address pair. For inbound packets, the converse of the transformation is performed. A table of special routes is available for connection to exchanges with restrictive source IP requirements (eg. EBS & Reuters).

SessionType

There are four possible session types. Some sessions (those pointed towards venue endpoints marked as DATA) begin as DATA sessions. All other sessions begin as LOGON.

• LOGON - uses message parser in logon mode, can transition to any of the other modes upon logon confirmation
• TAKER - uses message parser in taker mode
• MAKER - uses message parser in maker mode
• PASS TKR - uses message parser in taker password-change mode
• PASS MKR - uses message parser in maker password-change mode
• COPY - incoming dropcopy feed
• DATA - ignore all content

TCP Payload Replacement

Reflector controls risk by nullifying outbound order actions that would have breached risk limits if allowed to proceed. To block an illegal order action, Reflector's message parser overwrites part of the TCP stream. Once overwritten, any subsequent resends of the same segments of the TCP stream will be overwritten again. To preserve performance and simplicity of implementation, Reflector refrains from either inserting or deleting bytes from the underlying TCP stream.

TCP Session Log

The payload of monitored TCP sessions (everything except DATA) are logged. One line is written per direction; if any bytes are replaced, it will be the replacement that gets logged.

Segmentation & Acknowledgements

The message parsers must consume complete messages one at a time, and in order. However, the TCP protocol delivers only a stream of bytes, possibly segmented differently from the messages. This mismatch in segmentation means that sometimes, Reflector is required to store packets containing partial messages while manufacturing/manipulating TCP ACKs. TCP frames manufactured by
Reflector can be identified by their unusual IPv4ID and TTL values.

Session Termination

Reflector's parser can return error for a variety of reasons: logon fail, wrong protocol, sanity checks, etc. Some exogenous factors can also call for immediate termination: session timeout, disabled credential, UNPLUG mode, etc. The session store executes this directive by generating a pair of TCP RST frames for each affected session.

Broken Session

If Reflector receives a TCP packet that fits the following requirements, it swallows the packet and generates a TCP RST packet in the reverse direction

• TCP packet is not part of a BGP session
• TCP packet cannot be matched to an existing session
• TCP packet is not an outbound SYN packet

Ethernet Loop Detection

Reflector periodically emits a packet with a custom EtherType and other unique identications. If Reflector captures a packet it has emitted in the opposite direction, it will schedule an orderly shutdown.