OpenAPI

Plain WebSocket streams in Saxo Bank OpenAPI

Much of the information available through the Saxo Bank OpenAPI changes frequently. This is information such as quotes, positions, orders, balances etc. Having to poll this information with high frequency is not a viable solution. The IO overhead alone will makes such a solution undesirable. To solve this problem Saxo Bank OpenAPI offers up WebSocket streams for delivering such information, allowing the server to push information when it changes. The result is less IO usage and lower latency.

Setting up a WebSocket stream means setting up the stream and a number of subscriptions that tells the Saxo Bank OpenAPI what information you want updates on. Saxo Bank OpenAPI offers three modes of streaming. SignalR WebSocket streams, SignalR long polling and plain WebSocket streams. This document describes how to use the plain web socket streams.

In this document you will learn
- How to connect to the WebSocket server
- How to disconnect from the WebSocket server
- How to reconnect when the connection is lost
- How to understand and parse the messages you receive

It is important to understand the difference between WebSocket streams and the subscriptions that generate data for those streams, and how they work together. The WebSocket stream is the delivery channel for the updates you have asked for. The Subscriptions are where you ask for specific updates to be sent out. A stream can deliver data for multiple subscriptions. Streams are identified by a context id and subscriptions are identified by a reference id.

Connecting to the WebSocket server

To connect to the WebSocket server the client must go through the following steps:

  1. Obtain an access token 
    In order to create a WebSocket connection the client has to obtain an access token. Read more about how to obtain an access token.

  2. Create a contextId 
    The client must also create an identifier of the streaming connection, which is called the contextId. The context id must identify the streaming connection within the client session, so if the client has multiple streaming connections those connections must have different context ids.

  3. Initiate the WebSocket request 
    The client must then initiate the WebSocket connection handshake. Most of the protocol handling is normally performed by the WebSocket library, but a few points must be observed:
    • The context id must be specified as a query string parameter with the name contextId. It can be at most 50 characters (a-z, A-Z, -, and 0-9).
    • The access token must be sent in the WebSocket request. The access token can be sent either in the Authorization header or as part of the query string depending on what is supported by the client platform. The preferred method is to send the access token in the Authorization header.

  4. Create subscriptions
    In order to start receiving data you have to set up a subscription for the data you want. Subscriptions can be set up on a number of topics. 
    The server will buffer messages, so it is possible to create subscriptions and initiate the web socket request in parallel.
    To set up a subscription you need to provide a context id, reference id and an access token. 
    • The context id is the same context id used to initiate the web socket request and tells the API to send messages out on that web socket connection.
    • The reference id identifies messages from a specific subscription when they are sent out on a web socket connection. This enables you to set up multiple subscriptions on the same web socket connection.
    • The access token must be sent as an Authorization header in the POST request.

Here is an example connection request where the access token is sent as an HTTP header:

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com

And here is an example connection request where the access token is sent in the query string:

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection&authorization=BEARER%20[TOKEN]
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Host: streaming.saxobank.com


If the connection can be established the server responds with HTTP status 101 Switching Protocols according to the WebSocket protocol:

HTTP/1.1 101 Switching Protocols
Cache-Control: private
Upgrade: WebSocket
Server: Microsoft-IIS/8.5
Sec-WebSocket-Accept: fqGuSI/6utSRex2gWkBHfWGKDLo=
Connection: Upgrade

If the connection cannot be established the server responds with an HTTP status code different from 101 Switching Protocols. The following 4xx range codes may be returned from the WebSocket server:

HTTP statusPossible reasons
400 Bad Request

The context id is missing.

The specified context id is not correctly formatted.

401 Unauthorized

The access token was not sent as part of the connection request.

The access token has an invalid format.

The access token has expired.

409 ConflictThe client session already has a WebSocket connection with the same context id.
426 Upgrade RequiredThe WebSocket version is different from 13. All lower versions are preliminary, but a few older client libraries use them.
429 Too Many RequestsThe client session already has the allowed number of WebSocket connections. The number depends on the throttling profile assigned to the application.

How errors are handled in the client code typically depends on the WebSocket client library and the client code may not have direct access to the HTTP status code.

Subscription example

Here is an example of a subscription request. The example is a request for a price subscription. The request always has a ContextId and a ReferenceId along with some optional subscription parameters, and then a list of Arguments that are specific to a subscription.

When the subscription is successfully created a 201 response is returned and a Location header is set. This header is important to read and keep for the lifetime of the subscription. This is the location that should be used when deleting the subscription again.

POST https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com
{
  "Arguments": {
    "AssetType": "FxSpot",
    "Uic": 22
  },
  "ContextId": "20180712075219681",
  "ReferenceId": "IP44964",
  "RefreshRate": 1000
}

Receiving messages

When a WebSocket connection has been established the client can start receiving messages from subscriptions, for example price subscriptions.

Data messages are sent as a stream over a series of binary WebSocket frames. Each WebSocket frame can contain multiple data messages, but a data message can also be split over several WebSocket frames in which case a WebSocket continuation frame will follow. In the illustration below there are three WebSocket frames (A, B, and C) and four data messages (1, 2, 3, and 4). 

                 +----------------+----------------+----------------+
WebSocket frame: |     A    FIN=0 |     B    FIN=1 |      C   FIN=1 |
                 +---------+------+----------------+--------+-------+
Data message:    |    1    |           2           |   3    |   4   |
                 +---------+-----------------------+--------+-------+

The client receives the WebSocket frames one at a time and in order so the first frame received is A, then B is received, and finally C:

  • Frame A contains message 1 and the first part of message 2 so its FIN flag will be 0 (see the The WebSocket Protocol section 5.2) to indicate that more frames will follow to complete the message.
  • Frame B contains the remainder of message 2 and its FIN flag is 1 to indicate that the message is complete.
  • Frame C contains messages 3 and 4 and its FIN flag 1 to indicate that the message is complete.

The data messages indicate their own size so the client must parse the data message to identify the boundaries between messages. For example, the client receives frame C and notes that the total size of the WebSocket frame payload is 1000 bytes. It then reads message 3 according to the format specified below. Assume that message 3 is 600 bytes long. Since only 600 of the 1000 bytes have been read to complete message 3, the client should read another message from the same frame and keep on doing so until the whole frame has been read. In the example message 4 would be 400 bytes long.

Individual data messages have the following layout:

Byte indexSize in bytesDescription
08

Message identifier

64-bit little-endian unsigned integer identifying the message. The message identifier is used by clients when reconnecting. It may not be a sequence number and no interpretation of its meaning should be attempted at the client.

82

Reserved

This field is reserved for future use and it should be ignored by the client.

101

Reference id size (Srefid)

The number of characters/bytes in the reference id that follows.

11Srefid

Reference id

ASCII encoded reference id for identifying the subscription associated with the message. The reference id identifies the source subscription or type of control message.

11+Srefid1

Payload format

8-bit unsigned integer identifying the format of the message payload. Currently the following formats are defined:

0: The payload is a UTF-8 encoded text string containing JSON.

1: The payload is a binary protobuffer message.

The format is selected when the client sets up a streaming subscription so the streaming connection may deliver a mixture of message format. Control messages such as subscription resets are not bound to a specific subscription and are always sent in JSON format.

12+Srefid4

Payload size (Spayload)

64-bit little-endian unsigned integer indicating the size of the message payload.

16+SrefidSpayload

Payload

Binary message payload with the size indicated by the payload size field. The interpretation of the payload depends on the message format field.

Disconnecting

When the client wants to disconnect it will have to send a WebSocket close frame to the server. The server will then respond with a close frame of its own. Disconnecting will clean up your subscriptions on the server and free up connections and subscriptions, so they do not count against you when we throttle connections.  

Reconnecting

In case of errors that occur on the transport level, you will want to reconnect to the socket. To make sure this has as low an impact as possible the server keeps a small buffer of the latest messages sent to the client. When reconnecting you can specify the last message id you have received which will tell the server to start sending from that message. If you specify message id 10 as your last seen message, then the first message you will see after a reconnect is the message that comes after message 10. Do not assume that the message ids are ordered. They are merely ids and we can restart the sequence at any time if we need to. 

Example of a reconnection request where the last seen message id is provided in the query string parameter messageid.

GET https://streaming.saxobank.com/sim/openapi/streamingws/connect?contextId=MyConnection&messageid=10
Connection: Upgrade
Upgrade: WebSocket
Sec-WebSocket-Key: gnPAlQRoyFI3zMnCgm3vlQ==
Sec-WebSocket-Version: 13
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com

Resetting subscriptions

In the event that the Web Socket server detects a message loss or other error, it will send a Reset Subscription control message on the Web Socket connection. This message tells you that you have to recreate the subscriptions. The reset subscription messages may include a list of reference ids, that should be reset. If it does not include such a list or the list is empty, you will have to reset all subscriptions.
When resetting a subscription be sure to first delete the old subscriptions so these don't count against you when we throttle connections.

An example of a reset subscriptions control message is shown below. Like all control messages the reference id starts with an underscore. 

{
  "ReferenceId": "_resetsubscriptions",
  "Timestamp": "2018-07-19T11:47:41.841522Z",
  "TargetReferenceIds": [
    "IP44964"
  ]
}

There are two ways to delete a subscription and create a new one (delete must come first, to prevent reaching subscription limits).

The most efficient way is to issue a new subscription request, and use the ReplaceReferenceId field to remove the existing one. The backend will handle the delete request. This minimizes the window without an active subscription.

Alternatively, when you want to control this yourself (or if you want to replace multiple subscriptions by one new subscription, you can do a delete request:

To delete a subscription you will need to send a DELETE request to the location of the subscription resource. This location was returned in the Location header when creating the response and will be constructed by url where you subscribed, but with the context id and reference id appended as url segments (/{_contextId}/{_referenceId}) as shown in the example shown below.

DELETE https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions/20180712075219681/IP44964
Authorization: BEARER [TOKEN]

Then followed by issuing a new subscription request with a new resource id but the same context id as show below

POST https://gateway.saxobank.com/sim/openapi/trade/v1/prices/subscriptions
Authorization: BEARER [TOKEN]
Host: streaming.saxobank.com
{
  "Arguments": {
    "AssetType": "FxSpot",
    "Uic": 22
  },
  "ContextId": "20180712075219681",
  "ReferenceId": "IP55784",
  "ReplaceReferenceId": "IP44964",  // In case you skipped the DELETE request
  "RefreshRate": 1000
}

Re-authorizing

Tokens issued from our OAuth2 server only have a limited lifetime. Along with the access token we also issue a refresh token. You need this refresh token to get a new access token before the current one expires. With normal http requests, you would just replace your access token once it has been renewed and include the new token in all subsequent requests. But Web Socket Streams don't work like regular HTTP requests, so the server only knows the token you had when you initiated the streaming connection. Once you have refreshed your token, you will have to let the server know that you have a new and valid token. Otherwise the server will disconnect the streaming connection once the initial token expires.

Doing this is quite simple. You need to get a new access token from our OAuth2 server and then execute a PUT request with a correct Authorization header and a context id in the querystring (/streamingws/authorize?contextid={contextid}). The server will return a 202 Accepted status code if the new access token is valid. The server will return 202 even if the context ids do not match any current connections.

Example of a re-authorization request.

PUT https://streaming.saxobank.com/sim/openapi/streamingws/authorize?contextid=20180712075219681
Authorization: BEARER [TOKEN]

Control messages

In addition to the messages that you have chosen to subscribe to, the server might also need to send you messages that tell you about the state of the subscriptions you have set up. Some of these require actions from you, and other are just informational.

Common for all control messages is that their reference id always starts with an underscore. For now the control messages you can expect to receive are these

ReferenceIdExampleDescriptionAction Required by the client
_heartbeat
{
  "ReferenceId": "_heartbeat",
  "Heartbeats": [
    {
      "OriginatingReferenceId": "IP44964",
      "Reason": "NoNewData"
    }
  ]
}

Heartbeat messages

Heartbeat messages are sent when no data is sent for a while on a subscription, streaming server starts sending out heartbeats to let the client know that it is still active.

Heartbeats can be bundled for a connection, so you can get one for each of the subscriptions on that connection. The list of possible reasons for a heartbeat are:

  • NoNewData
    This reason indicates that there just isn't any new data available, but we are still alive and will send out data when something is available.
  • SubscriptionTemporarilyDisabled
    No data is currently being sent due to a recoverable error on the server, e.g. failure to connect to a database or internal service. The client should accept that data is not available at the expected rate and disable functions accordingly until data is available again. The subscription will continue to deliver heartbeats until data is available again.
  • SubscriptionPermanentlyDisabled
    No data will be available on the subscription. Most likely this is due to a misconfiguration. The client should not attempt to reset the subscription and should not expect any data on it. The subscription will continue to deliver heartbeats until the subscription is removed.

Keep track of when you last received a message or a heartbeat message. If you stop receiving messages you should reconnect and tell the server what the last message id you saw was. [ADD APPROPRIATE CLIENT TIMEOUT]

If the heartbeat reason is SubscriptionPermanentlyDisabled you will need to remove the subscriptions defined by the OriginatingReferenceId.

_disconnect
{
  "ReferenceId": "_disconnect"
}

Disconnect messages

It is possible that a subscription will be closed by the server. This can happen if a user changes their account password for example. When this happens a "_disconnect" instruction will be sent out on the streams opened by that user and after that no more messages will be sent on those connections. The client cannot create a new subscription after receiving a disconnect message because they have been logged out. All subsequent requests will be rejected with a 401 - Unauthorized response from the server until the user is logged out and logs back in. The message will be sent to all open connections at the same time.

Prompt user to log in again and after that recreate the WebSocket connection and set up subscriptions again.
_resetsubscriptions
{
  "ReferenceId": "_resetsubscriptions",
  "TargetReferenceIds": [
    "IP44964"
  ]
}

Subscription reset messages

In some error scenarios, the streaming server may send subscription reset messages to instruct the client to reset a given subset of its subscriptions. In this case, value of ReferenceID in the stream data will be "_resetsubscriptions" and reference ids of the subscriptions which should be reset is sent in parameter TargetReferenceIds. If the list is empty you should reset all subscriptions.

Resetting a subscription means that the subscription should be deleted and another one created in its place using a new reference id. Any update received for a subscription after a reset instruction must be ignored.

Delete subscriptions specified by the array of reference ids in the TargetReferenceIds array.

To delete a subscription you will need to send a DELETE request to the location of the subscription resource. This location was returned in the Location header when creating the response.

Create new subscriptions to replace the old subscriptions and start from new by applying updates using the snapshot received from these new subscriptions.

What state do I need to keep

When consuming our streaming services there is not much state you need to keep, but there is still some.

The first thing you need to keep track of are the ids. Your WebSocket connection is defined by a context id and messages are identified by a message id and a reference id. You will need these to effectively route messages to the right place when you receive them. The messages are composed in a way that enables you to route messages without deserializing them first. You can deserialize the header without deserializing the payload.

You also need to keep track of the latest message id you have received. In case you need to reconnect to the WebSocket you will need to pass this to the WebSocket server in order to pick up messages where you left off.

The messages you receive are only updates. So you need to store the original snapshot with all received updates applied. The messages you receive are ordered and need to be applied to the snapshot in that order.

Updates

The messages you receive from the WebSocket stream are not complete messages. The complete messages can be very big, and in order to optimize the network utilization all messages are delta compressed. This means that only the properties whose values have changed are contained in the messages you receive. To assemble the complete message you will need to apply the update received to the original snapshot. 

Because messages are are only updates it is important that you preserve the order in which you have received the messages, and that you apply deltas in that order. If you change the order, the message will become corrupt and you will not be able to trust that the state is correct.
It is also a likely scenario that updates will arrive before the snapshot. In that case it is important to queue up messages and apply them when the snapshot arrives. If not all updates are applied the message will become corrupt and you will not be able to trust that the state is correct.

Do not assume that the message ids are ordered. They are arbitrary ids and we can restart the sequence at any time if we need to.