MarketFactory recommends using the RealLogic SBETool. It should be noted that we use v1.25.1 to avoid a bug in the CME ilink3 schema that is revealed with later versions of the tool, but clients are free to use later versions.
The breaking change was introduced in SBE Tool v1.25.2, as described in RealLogic issues #889 and #917: because it is impossible to specify a character value of 0x00 in a valid XML document, the proposed solution, such as it is, is to convince CME to remove the null_value="0" attribute from the specification of charNULL in their schema. This may take a while.
When directly inspecting "on-the-wire" representations of char datatypes - including enumerations which may take numeric values - be aware that these are subject to UTF-8 encoding, and you may be inspecting the decimal code of the given character.
Refer to https://en.wikipedia.org/wiki/UTF-8#Codepage_layout for details.
<enum name="Side" description="Side" encodingType="char">
<validValue name="Buy">1</validValue>
<validValue name="Sell">2</validValue>
<validValue name="TwoWay">7</validValue>
</enum> |
You may observe the following values:
The string with either be null-terminated or will be of the size of the fixed length field. So if the string is smaller than the field length; it will end with a \0. If the string is the same length as the field length then there is no terminator.
When writing you do not have to ensure the remainder of the array is all \0 values - although that would work. The string just needs to be terminated with a single \0; or if the string is exactly as large as the field length - you don't need to do anything.
The server expects string fields to be terminated with a \0.
It's worth remembering that the FIX protocol checksum is at the application level. The underlying TCP transport also incorporates it's own CRC and checksum checks.
The main reason this application checksum exists (in my opinion at least) is that the FIX protocol is a simple structured string and there is a real chance of it being mangled at the application level (not all FIX 'engines' are equal), which those transport-level checks would not protect against.
For SBE, code stubs are generated against the schema, using these to read/write a given message or field ensures it is well-formed. There are still ways to break the SBE messaging - e.g. receiving an enum value that is not supported in your stubs (i.e. schema compatibility issues). Any such error on reading/writing is a disconnection event.
In addition, SBE is a performance-oriented wire protocol. If checksums are included as for the FIX protocol, application-level checksum computation becomes a significant component of the overall cost.
Unlike FIXML, there is no definition of an attributeGroup component that is defined once and referenced multiple times (e.g. in a complexType). In SBE each group has to be an explicit component of it's parent group or message. To minimise the amount of 'duplicate' code, there are two broad avenues to consider:
There is a way to aggregate multiple fields together as a composite datatype (and a composite may itself reference other composite types), so using our NoHops group as an example, in theory we could move to something like:
<type name="UTCTimestamp" presence="optional" nullValue="0" primitiveType="uint64" semanticType="UTCTimestamp"/>
<type name="SeqNum" nullValue="0" presence="optional" nullValue="0" primitiveType="uint32" semanticType="SeqNum"/>
<composite name="NoHopsFieldsType">
<ref name="HopRefID" type="SeqNum"/>
<ref name="HopSendingTime" type="UTCTimestamp"/>
<ref name="HopNetworkTime" type="UTCTimestamp"/>
<ref name="HopArrivalTime" type="UTCTimestamp"/>
</composite>
<sbe:message name=MyMessage id=1">
...
<group name="NoHops" id="627" dimensionType="groupSizeEncoding">
<field name="NoHopsFields" id="20000" type=" NoHopsFieldsType"/>
</group>
</sbe:message>
|
Extending this idea, we could specify every single field/type as a composite type containing a single type definition, allowing us to create composites (with refs for the individual field type definitions as above) for each unique body or group level block. But moving from multiple fields to compound types is not free and things become very awkward in other ways:
In short, this is not a fruitful path. The "SBE way" is for each message to be self-contained, if you didn't want to rely on the generated stubs (with the duplicate types etc), then at a slight performance hit, the typical alternative would be to use an on-the-fly decoder model, which requires usage of the SBE intermediate representation to process each message in a more generic manner.
MarketFactory has thought about creating a 'smarter' stub generator that would identify common groups, types etc and generate a more optimised set of getter/setter etc primitives. This is conceptually doable and probably much more in line with what you're looking for, although this work is not currently planned.
A typical technique is to write every outgoing message to a memory mapped file at the same time as sending it down the TCP socket, for possible retrieval later. The file could be persisted to disk periodically if there was a need to keep it longer term.
If you don't need to replay messages to the venue, then just storing the sequence number is good enough.