Plain WebSocket Streaming

Plain WebSocket streams in Saxo Bank OpenAPI

Much of the information available through the Saxo Bank OpenAPI changes frequently. This is information such as quotes, positions, orders, balances etc. Having to poll this information with high frequency is not a viable solution. The IO overhead alone will makes such a solution undesirable. To solve this problem Saxo Bank OpenAPI offers up WebSocket streams for delivering such information, allowing the server to push information when it changes. The result is less IO usage and lower latency.

Setting up a WebSocket stream means setting up the stream and a number of subscriptions that tells the Saxo Bank OpenAPI what information you want updates on. Saxo Bank OpenAPI offers three modes of streaming. SignalR WebSocket streams, SignalR long polling and plain WebSocket streams. This document describes how to use the plain web socket streams.

In this document you will learn
- How to connect to the WebSocket server
- How to disconnect from the WebSocket server
- How to reconnect when the connection is lost
- How to understand and parse the messages you receive

It is important to understand the difference between WebSocket streams and the subscriptions that generate data for those streams, and how they work together. The WebSocket stream is the delivery channel for the updates you have asked for. The Subscriptions are where you ask for specific updates to be sent out. A stream can deliver data for multiple subscriptions. Streams are identified by a context id and subscriptions are identified by a reference id.

Connecting to the WebSocket server

To connect to the WebSocket server the client must go through the following steps:

1. Obtain an access token 
   In order to create a WebSocket connection the client has to obtain an access token. Read more about how to obtain an access token.
   
   
2. Create a contextId 
   The client must also create an identifier of the streaming connection, which is called the contextId. The context id must identify the streaming connection within the client session, so if the client has multiple streaming connections those connections must have different context ids.
   
   
3. Initiate the WebSocket request 
   The client must then initiate the WebSocket connection handshake. Most of the protocol handling is normally performed by the WebSocket library, but a few points must be observed:
   • The context id must be specified as a query string parameter with the name contextId. It can be at most 50 characters (a-z, A-Z, -, and 0-9).
   • The access token must be sent in the WebSocket request. The access token can be sent either in the Authorization header or as part of the query string depending on what is supported by the client platform. The preferred method is to send the access token in the Authorization header.
     
     
4. Create subscriptions
   In order to start receiving data you have to set up a subscription for the data you want. Subscriptions can be set up on a number of topics. 
   The server will buffer messages, so it is possible to create subscriptions and initiate the web socket request in parallel.
   To set up a subscription you need to provide a context id, reference id and an access token. 
   • The context id is the same context id used to initiate the web socket request and tells the API to send messages out on that web socket connection.
   • The reference id identifies messages from a specific subscription when they are sent out on a web socket connection. This enables you to set up multiple subscriptions on the same web socket connection.
   • The access token must be sent as an Authorization header in the POST request.

Here is an example connection request where the access token is sent as an HTTP header:

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com

And here is an example connection request where the access token is sent in the query string:

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection&authorization=BEARER%20[TOKEN]
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Host: streaming.saxobank.com

If the connection can be established the server responds with HTTP status 101 Switching Protocols according to the WebSocket protocol:

HTTP/1.1 101 Switching Protocols
Cache-Control: private
Upgrade: WebSocket
Server: Microsoft-IIS/8.5
Sec-WebSocket-Accept: fqGuSI/6utSRex2gWkBHfWGKDLo=
Connection: Upgrade

If the connection cannot be established the server responds with an HTTP status code different from 101 Switching Protocols. The following 4xx range codes may be returned from the WebSocket server:

How errors are handled in the client code typically depends on the WebSocket client library and the client code may not have direct access to the HTTP status code.

Subscription example

Here is an example of a subscription request. The example is a request for a price subscription. The request always has a ContextId and a ReferenceId along with some optional subscription parameters, and then a list of Arguments that are specific to a subscription.

When the subscription is successfully created a 201 response is returned and a Location header is set. This header is important to read and keep for the lifetime of the subscription. This is the location that should be used when deleting the subscription again.

POST https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com
{
  "Arguments": {
    "AssetType": "FxSpot",
    "Uic": 22
  },
  "ContextId": "20180712075219681",
  "ReferenceId": "IP44964",
  "RefreshRate": 1000
}

Receiving messages

When a WebSocket connection has been established the client can start receiving messages from subscriptions, for example price subscriptions.

Data messages are sent as a stream over a series of binary WebSocket frames. Each WebSocket frame can contain multiple data messages, but a data message can also be split over several WebSocket frames in which case a WebSocket continuation frame will follow. In the illustration below there are three WebSocket frames (A, B, and C) and four data messages (1, 2, 3, and 4). 

                 +----------------+----------------+----------------+
WebSocket frame: |     A    FIN=0 |     B    FIN=1 |      C   FIN=1 |
                 +---------+------+----------------+--------+-------+
Data message:    |    1    |           2           |   3    |   4   |
                 +---------+-----------------------+--------+-------+

The client receives the WebSocket frames one at a time and in order so the first frame received is A, then B is received, and finally C:

• Frame A contains message 1 and the first part of message 2 so its FIN flag will be 0 (see the The WebSocket Protocol section 5.2) to indicate that more frames will follow to complete the message.
• Frame B contains the remainder of message 2 and its FIN flag is 1 to indicate that the message is complete.
• Frame C contains messages 3 and 4 and its FIN flag 1 to indicate that the message is complete.

The data messages indicate their own size so the client must parse the data message to identify the boundaries between messages. For example, the client receives frame C and notes that the total size of the WebSocket frame payload is 1000 bytes. It then reads message 3 according to the format specified below. Assume that message 3 is 600 bytes long. Since only 600 of the 1000 bytes have been read to complete message 3, the client should read another message from the same frame and keep on doing so until the whole frame has been read. In the example message 4 would be 400 bytes long.

Individual data messages have the following layout:

Disconnecting

When the client wants to disconnect it will have to send a WebSocket close frame to the server. The server will then respond with a close frame of its own. Disconnecting will clean up your subscriptions on the server and free up connections and subscriptions, so they do not count against you when we throttle connections.  

Reconnecting

In case of errors that occur on the transport level, you will want to reconnect to the socket. To make sure this has as low an impact as possible the server keeps a small buffer of the latest messages sent to the client. When reconnecting you can specify the last message id you have received which will tell the server to start sending from that message. If you specify message id 10 as your last seen message, then the first message you will see after a reconnect is the message that comes after message 10. Do not assume that the message ids are ordered. They are merely ids and we can restart the sequence at any time if we need to. 

Example of a reconnection request where the last seen message id is provided in the query string parameter messageid.

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection&messageid=10
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com

Resetting subscriptions

In the event that the Web Socket server detects a message loss or other error, it will send a Reset Subscription control message on the Web Socket connection. This message tells you that you have to recreate the subscriptions. The reset subscription messages may include a list of reference ids, that should be reset. If it does not include such a list or the list is empty, you will have to reset all subscriptions.
When resetting a subscription be sure to first delete the old subscriptions so these don't count against you when we throttle connections.

An example of a reset subscriptions control message is shown below. Like all control messages the reference id starts with an underscore. 

{
  "ReferenceId": "_resetsubscriptions",
  "Timestamp": "2018-07-19T11:47:41.841522Z",
  "TargetReferenceIds": [
    "IP44964"
  ]
}

There are two ways to delete a subscription and create a new one (delete must come first, to prevent reaching subscription limits).

The most efficient way is to issue a new subscription request, and use the ReplaceReferenceId field to remove the existing one. The backend will handle the delete request. This minimizes the window without an active subscription.

Alternatively, when you want to control this yourself (or if you want to replace multiple subscriptions by one new subscription, you can do a delete request:

To delete a subscription you will need to send a DELETE request to the location of the subscription resource. This location was returned in the Location header when creating the response and will be constructed by url where you subscribed, but with the context id and reference id appended as url segments (/{_contextId}/{_referenceId}) as shown in the example shown below.

DELETE https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions/20180712075219681/IP44964
Authorization: BEARER [TOKEN]

Then followed by issuing a new subscription request with a new resource id but the same context id as show below

POST https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com
{
  "Arguments": {
    "AssetType": "FxSpot",
    "Uic": 22
  },
  "ContextId": "20180712075219681",
  "ReferenceId": "IP55784",
  "ReplaceReferenceId": "IP44964",  // In case you skipped the DELETE request
  "RefreshRate": 1000
}

Re-authorizing

Tokens issued from our OAuth2 server only have a limited lifetime. Along with the access token we also issue a refresh token. You need this refresh token to get a new access token before the current one expires. With normal http requests, you would just replace your access token once it has been renewed and include the new token in all subsequent requests. But Web Socket Streams don't work like regular HTTP requests, so the server only knows the token you had when you initiated the streaming connection. Once you have refreshed your token, you will have to let the server know that you have a new and valid token. Otherwise the server will disconnect the streaming connection once the initial token expires.

Doing this is quite simple. You need to get a new access token from our OAuth2 server and then execute a PUT request with a correct Authorization header and a context id in the querystring (/streamingws/authorize?contextid={contextid}). The server will return a 202 Accepted status code if the new access token is valid. The server will return 202 even if the context ids do not match any current connections.

Example of a re-authorization request.

PUT https://streaming.saxobank.com/sim/openapi/streamingws/authorize?contextid=20180712075219681
Authorization: BEARER [TOKEN]

Control messages

In addition to the messages that you have chosen to subscribe to, the server might also need to send you messages that tell you about the state of the subscriptions you have set up. Some of these require actions from you, and other are just informational.

Common for all control messages is that their reference id always starts with an underscore. For now the control messages you can expect to receive are these

{
  "ReferenceId": "_heartbeat",
  "Heartbeats": [
    {
      "OriginatingReferenceId": "IP44964",
      "Reason": "NoNewData"
    }
  ]
}

{
  "ReferenceId": "_disconnect"
}

{
  "ReferenceId": "_resetsubscriptions",
  "TargetReferenceIds": [
    "IP44964"
  ]
}

What state do I need to keep

When consuming our streaming services there is not much state you need to keep, but there is still some.

The first thing you need to keep track of are the ids. Your WebSocket connection is defined by a context id and messages are identified by a message id and a reference id. You will need these to effectively route messages to the right place when you receive them. The messages are composed in a way that enables you to route messages without deserializing them first. You can deserialize the header without deserializing the payload.

You also need to keep track of the latest message id you have received. In case you need to reconnect to the WebSocket you will need to pass this to the WebSocket server in order to pick up messages where you left off.

The messages you receive are only updates. So you need to store the original snapshot with all received updates applied. The messages you receive are ordered and need to be applied to the snapshot in that order.

Updates

The messages you receive from the WebSocket stream are not complete messages. The complete messages can be very big, and in order to optimize the network utilization all messages are delta compressed. This means that only the properties whose values have changed are contained in the messages you receive. To assemble the complete message you will need to apply the update received to the original snapshot. 

Because messages are are only updates it is important that you preserve the order in which you have received the messages, and that you apply deltas in that order. If you change the order, the message will become corrupt and you will not be able to trust that the state is correct.
It is also a likely scenario that updates will arrive before the snapshot. In that case it is important to queue up messages and apply them when the snapshot arrives. If not all updates are applied the message will become corrupt and you will not be able to trust that the state is correct.

Do not assume that the message ids are ordered. They are arbitrary ids and we can restart the sequence at any time if we need to.