RFA Dictionary Failure Probabilities, Direct File Download from LSEG, and Serialising In-Memory Dict

mktdata

Subject:- RFA Dictionary — Failure Probabilities, Direct File Download from LSEG, and Serialising In-Memory Dictionary to Disk.

Dear LSEG Support Team,

We are using the RFA C++ API to consume real-time market data via a TREP/ADS infrastructure. Our current implementation downloads the RWFFld (RDMFieldDictionary) and RWFEnum (EnumType) dictionaries from the TREP/ADS provider at connection time and populates them directly into in-memory dictionary objects same as RFA Consumer Client Application. We have a local file fallback, but maintaining up-to-date local files manually is operationally fragile.

We have below questions about this particular RFA Dictionary Download domain

Section 1 — Expected Frequency of Dictionary Download Failure Scenarios

During our implementation and code review we identified the following categories of failure. We would appreciate your guidance on how commonly each occurs in production TREP/ADS environments, and whether each is typically transient or indicative of a persistent misconfiguration:

1a. Directory Request Registration Failures (prerequisite before any dictionary download can begin):

• rfa::common::Exception from registerClient() when registering the MMT_DIRECTORY subscription
• Standard or unknown C++ exception during SendRequest() for the directory

1b. Directory Response Failures:

• OMMCmdErrorEventEnum received — malformed directory request or unrecognised service name
• Received GenericMsg or a non-MMT_DIRECTORY model type in directory response
• Unexpected event type (!= OMMItemEventEnum) on the directory stream
• Refresh complete but stream state is not OpenEnum — directory remains unavailable
• Blank payload in a directory refresh response
• Payload data type is not MapEnum
• Service name key in map entry is not DataBufferEnum
• Service entry data is not FilterListEnum
• SERVICE_INFO_ID filter entry is not ElementListEnum
• Service name in SERVICE_INFO_ID does not match the map entry key
• SERVICE_STATE_ID filter entry is not ElementListEnum
• SERVICE_GROUP_ID filter entry is not ElementListEnum
• Group ID length is not a multiple of 2 bytes
• SERVICE_LOAD_ID filter entry is not ElementListEnum
• SERVICE_DATA_ID filter entry is not ElementListEnum
• SERVICE_LINK_ID filter entry is not MapEnum or its MapEntry is not ElementListEnum

1c. Directory Status Message Failures:

• SourceUnknownEnum status code — service name not recognised by the provider
• Stream state OpenEnum with data state SuspectEnum — data may be stale, directory marked unavailable
• Stream state ClosedEnum or ClosedRecoverEnum — directory stream closed, market data unavailable

1d. Dictionary Request Registration Failures:

• rfa::common::Exception from registerClient() for field or enum dictionary (RWFFld / RWFEnum)
• Standard or unknown C++ exception during SendRequest() for dictionary

1e. Dictionary Response / Message Type Failures:

• Received GenericMsg or non-MMT_DICTIONARY model type in dictionary response
• Received Update message instead of Refresh for the dictionary stream
• Unexpected event type (!= OMMItemEventEnum) on the dictionary stream

1f. Dictionary Refresh Message Processing Failures:

• DoNotCacheFlag set on dictionary refresh — message skipped, dictionary not populated
• No PayloadFlag on the refresh response — payload absent
• Response handle does not match either m_rdmFldDictHandle or m_rdmEnumDictHandle

1g. Dictionary Type Consistency Failures:

• Mismatch between TypeInTheHandle, TypeInSummaryData, and TypeInAttribInfo
• Dictionary type in payload does not match the requested type
• Dictionary type in header does not match the requested type
• Service name in AttribInfo does not match configured service name — dictionary silently not populated

1h. Dictionary Payload Decoding Failures:

• Payload data type is not SeriesEnum
• Both dictionaries processed (m_counterDictionaryTypesProcessed == 2) but neither marked complete

1i. Stream Status Failures on the Dictionary Stream:

• ClosedEnum — permanently closed, cannot recover via re-request
• ClosedRecoverEnum — triggers version check (DICTIONARY_INFO) or full re-download; temporarily unavailable
• SuspectEnum data state — indicates a new dictionary version is available; full re-request issued

For each of the above, we would like to understand:

• Under what real-world conditions does this typically occur?
• Is it transient (e.g. network hiccup, ADS restart) or a persistent misconfiguration?
• Is there a recommended detection and recovery strategy beyond falling back to local files?
  Is these cases occurs frequently?
  Is the particular service goes down frequently?

Section 2 — Direct File Download from LSEG

Is it possible to download the RDMFieldDictionary and enumtype.def files directly from an LSEG-hosted server or distribution endpoint, independently of the TREP/ADS infrastructure? Specifically:

• Does LSEG provide any HTTP/SFTP endpoint, package feed, or SDK distribution mechanism from which clients can pull the latest dictionary files directly?
• Would files obtained this way be guaranteed to be consistent with what the connected TREP/ADS provider serves over the RFA protocol stream?

Section 3 — Serialising In-Memory Dictionary Back to Disk

Once the in-memory dictionary is successfully populated from the TREP/ADS stream, we would like to persist it to disk as RDMFieldDictionary and enumtype.def files. This would give us a reliable, always up-to-date local fallback at the next startup, eliminating the dependency on manually maintained local copies. Is this feasible using the RFA API?

• Does rfa::data::FieldDictionary (or any related RFA class) expose a method to write or serialise the in-memory dictionary back to a file in the standard RDMFieldDictionary or enumtype format?
• If no such API exists, is there a supported RFA utility or alternative mechanism to achieve this?
• Are there any licensing or redistribution restrictions that would prevent us from persisting the dictionary content received over the wire back to disk in this way?

Section 4 — Is Persisting Dictionary Files to Disk Actually Necessary?

Our current implementation keeps both RWFFld and RWFEnum dictionaries entirely in memory after a successful download from TREP/ADS. On ClosedRecoverEnum we first issue a lightweight DICTIONARY_INFO version check, and only trigger a full re-download if the version has actually changed — avoiding redundant bandwidth usage. On SuspectEnum we reissue the full request immediately as the RDM guide requires.

Given this, we would like to understand:

• Is an in-memory dictionary sufficient for a production long-running consumer? Are there known scenarios — beyond process restart — where an in-memory dictionary becomes stale or invalid and cannot be recovered purely through re-requesting from TREP/ADS?
• At process startup, if the TREP/ADS provider is temporarily unavailable, we currently fall back to local dictionary files. If we were confident the TREP/ADS provider would always be available within a reasonable retry window at startup, would LSEG consider it safe to remove the local file fallback entirely and rely solely on the network download?
• Are there any LSEG-recommended patterns for clients that want zero dependency on locally installed dictionary files — for example, retrying the dictionary request with backoff until the provider comes up, rather than failing over to a static file?
• Does the SuspectEnum data state guarantee that the in-memory dictionary is still valid enough to continue decoding existing subscriptions while the re-request is in flight, or must we pause all market data decoding until the new dictionary is fully downloaded?

The goal is to determine whether local dictionary files serve any essential production purpose beyond startup resilience, or whether a robust in-memory approach with version-aware reconnect logic is the recommended long-term strategy.

We would appreciate any documentation pointers, best practice guides, or direct guidance your team can provide. Please let us know if you need further details about our environment or RFA version.

Jirapongse

Thank you for reaching out to us.

Section 1 — Expected Frequency of Dictionary Download Failure Scenarios

The failure cases identified in your review are primarily defensive validation checks and are not scenarios that would normally be encountered in a healthy ADS environment with a thoroughly tested RFA application. If the application receives invalid dictionary responses, the issue may originate from the application itself, the network, or the server infrastructure, and should be investigated by the API support team to determine the root cause.

Section 2 — Direct File Download from LSEG

Currently, we do not provide dedicated servers specifically for dictionary distribution. RFA applications must obtain dictionaries either from local files or from a market data infrastructure component, such as ADS, that supports the Dictionary Domain.

Section 3 — Serialising In-Memory Dictionary Back to Disk

I checked and found that the RDMFieldDictionary doesn't directly support this. I found this discussion. According to the answer, you can save the raw buffer in a file and then read it back. I am not sure if this method can be used with the RDMFieldDictionary. Regarding the license and redistribution, please contact your LSEG account team or sales team.

Section 4 — Is Persisting Dictionary Files to Disk Actually Necessary?

This current implementation looks fine for me. In RFA, an application can create multiple instances of RDMFieldDictionary. For example, it may maintain one dictionary loaded from local files and another downloaded from a server, such as ADS, then use whichever dictionary is valid or has the newer version. The data dictionary is still valid although the data stream is stale.