Broker API HubMenu

Stage 4 of 4, lesson 23 of 31. Improver

Handle an error reply: read it, then retry, sign in or stop

Fetch an expired session and a rate limit refusal from the practice API, read what each body says and act on it: retry only what can succeed on a second try, a fixed number of times.

Independent and unofficialCode run

Risk warning: CFDs and spread bets are complex instruments and come with a high risk of losing money rapidly due to leverage. You can lose money on these products, and each provider publishes its own loss figure under FCA rules. You should consider whether you understand how they work and whether you can afford to take the high risk of losing your money. Automated and API trading can amplify losses through bugs, connectivity failures and runaway orders.

You need: The lessons on retry and backoff and on staying under a rate limit. An internet connection. No broker account. No key.

The retry lesson used a pretend API that raised errors you had written yourself. A real API refuses with a reply: a status number and a body that says what went wrong. This lesson reads real replies of that kind and decides what to do about each.

Three functions keep the three jobs apart. fetch makes one request and returns the status and the body, whatever they are. decide looks at them and names an action. call runs the loop: try, decide, and try again only if the action was wait and retry.

decide is the rule to learn. A 429 means you asked too often, so waiting helps, and the body says for how long. A 500, 502, 503 or 504 is the server's own trouble and may pass. A 401 means your session has run out: asking again will be refused again until you sign in again. Anything else in the 400s means the request is wrong, and the same request will be refused for as long as you send it. Retrying a refusal that cannot change only uses up your allowance and hides the real fault.

Two things here are particular to the practice API. It is a folder of fixed files, so it answers every request with status 200 and puts the status a real server would send inside the body. fetch handles both: a real refusal, which urllib raises as an HTTPError, and the practice kind. And the rate limit file never relents, so the lesson only pretends to wait the thirty seconds it asks for. Your own program should really wait, and the pacing lesson shows how not to be refused in the first place.

The check at the end asserts how many attempts each address was given: one, one and three. Count your attempts in your own programs too. A retry loop nobody counted is the usual cause of a rate limit.

The practice API this lesson calls

Practice data, not any broker's API: fixed files of invented values on this site. Each opens in a browser tab, so you can read what the script will be sent.

Every address on the practice API, and what it cannot do

The code

# Read what an API says when it refuses, and decide what to do: carry on, wait and retry, or stop.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key. Its example errors are bodies to practise on.
import json
import os
import time
import urllib.error
import urllib.request

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1            # seconds to leave between calls
MAX_ATTEMPTS = 3     # how many times one request may be tried in all


def fetch(path):
    """One request. Returns the status and the body, whether the server was pleased or not."""
    request = urllib.request.Request(
        BASE_URL + path,
        headers={"Accept": "application/json", "User-Agent": "handle-an-error-reply-lesson"},
    )
    try:
        with urllib.request.urlopen(request, timeout=10) as reply:
            status, body = reply.status, json.load(reply)
    except urllib.error.HTTPError as refusal:
        # A real API sends a refusal with a status of 400 or more, and urllib raises it as an error.
        # The error still carries the status and a body. This assumes the body is JSON.
        return refusal.code, json.load(refusal)
    # The practice API cannot send any status but 200. It puts the status a real server
    # would have sent inside the body, so that is the one we act on.
    if "error" in body:
        status = body["error"]["status"]
    return status, body


def decide(status, body):
    """What to do about one reply. Returns an action and, for a retry, how many seconds to wait."""
    if status < 400:
        return "use it", 0
    error = body["error"]
    if status == 429:
        return "wait and retry", error.get("retryAfterSeconds", 5)
    if status in (500, 502, 503, 504):
        return "wait and retry", 5        # the server's own trouble, which may pass
    if status == 401:
        return "sign in again", 0         # a new session is needed. Asking again will not make one
    return "stop and fix the request", 0  # 400, 403, 404, 422: the same request will be refused again


def call(path, wait=time.sleep):
    """Try a request up to MAX_ATTEMPTS times. Only an answer of wait and retry earns another try."""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        status, body = fetch(path)
        action, seconds = decide(status, body)
        if action == "use it":
            print("  attempt", attempt, "status", status, "worked")
            return action, attempt, body
        print("  attempt", attempt, "status", status, body["error"]["code"] + ":", action)
        if action != "wait and retry":
            return action, attempt, body
        if attempt < MAX_ATTEMPTS:
            wait(seconds)
    return "gave up", MAX_ATTEMPTS, body


def pretend_wait(seconds):
    # The practice file asks for 30 seconds and never changes its mind, so the lesson does not
    # sit through them. A real program would call time.sleep(seconds) here.
    print("    told to wait", seconds, "seconds. A real program would sleep here.")
    time.sleep(PAUSE)


results = {}
for path in ["/account.json", "/errors/session-expired.json", "/errors/rate-limited.json"]:
    print("GET", path)
    action, attempts, body = call(path, wait=pretend_wait)
    results[path] = (action, attempts)
    print("  outcome:", action, "after", attempts, "attempt" if attempts == 1 else "attempts")
    time.sleep(PAUSE)

# The check. Each kind of reply was handled in its own way, and nothing was retried without cause.
assert results["/account.json"] == ("use it", 1)
assert results["/errors/session-expired.json"] == ("sign in again", 1)
assert results["/errors/rate-limited.json"] == ("gave up", MAX_ATTEMPTS)
print("Check passed: one reply used, one stopped at once, one tried", MAX_ATTEMPTS, "times and no more.")

What you should see

GET /account.json
  attempt 1 status 200 worked
  outcome: use it after 1 attempt
GET /errors/session-expired.json
  attempt 1 status 401 session_expired: sign in again
  outcome: sign in again after 1 attempt
GET /errors/rate-limited.json
  attempt 1 status 429 rate_limited: wait and retry
    told to wait 30 seconds. A real program would sleep here.
  attempt 2 status 429 rate_limited: wait and retry
    told to wait 30 seconds. A real program would sleep here.
  attempt 3 status 429 rate_limited: wait and retry
  outcome: gave up after 3 attempts
Check passed: one reply used, one stopped at once, one tried 3 times and no more.

Run with Python 3.12 on Windows against a copy of the practice API files served over HTTP on the same machine, with PRACTICE_API set to the address of the copy. They are the files this site serves. No account and no key.

Check yourself

Answer each question from the code above before you open it. The answers were written from the script on this page and what it printed.

Why was the expired session tried once and the rate limit three times?

decide returns sign in again for a 401 and wait and retry for a 429. call returns at once for any action other than wait and retry, and goes round again for that one until MAX_ATTEMPTS, which is 3.

The practice API answers every request with status 200. Where does the 401 in the output come from?

From the body. When the body has an error object, fetch replaces the status with error.status, which is the status a real server would have sent.

Where does the figure of 30 seconds come from?

From retryAfterSeconds in the body of the rate limit reply. decide reads it with error.get and falls back to 5 if a reply does not carry one.

Exercise

Add /errors/validation.json to the addresses the script tries. It is a 400, so it should be tried once and stopped. Then, under the outcome line, print each field the body says to correct, with the problem and the value that was received.

What the changed script prints
GET /account.json
  attempt 1 status 200 worked
  outcome: use it after 1 attempt
GET /errors/session-expired.json
  attempt 1 status 401 session_expired: sign in again
  outcome: sign in again after 1 attempt
GET /errors/rate-limited.json
  attempt 1 status 429 rate_limited: wait and retry
    told to wait 30 seconds. A real program would sleep here.
  attempt 2 status 429 rate_limited: wait and retry
    told to wait 30 seconds. A real program would sleep here.
  attempt 3 status 429 rate_limited: wait and retry
  outcome: gave up after 3 attempts
GET /errors/validation.json
  attempt 1 status 400 validation_failed: stop and fix the request
  outcome: stop and fix the request after 1 attempt
    to correct: size must be a number greater than 0, received 'two'
    to correct: direction must be BUY or SELL, received 'LONG'
Check passed: one reply used, one stopped at once, one tried 3 times and no more.

Captured from a run of the solution on . Run with Python 3.12 on Windows against a copy of the practice API files served over HTTP on the same machine, with PRACTICE_API set to the address of the copy. They are the files this site serves. No account and no key.

One solution
# Exercise solution: the lesson script with the one change the exercise asks for.
# Read what an API says when it refuses, and decide what to do: carry on, wait and retry, or stop.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key. Its example errors are bodies to practise on.
import json
import os
import time
import urllib.error
import urllib.request

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1            # seconds to leave between calls
MAX_ATTEMPTS = 3     # how many times one request may be tried in all


def fetch(path):
    """One request. Returns the status and the body, whether the server was pleased or not."""
    request = urllib.request.Request(
        BASE_URL + path,
        headers={"Accept": "application/json", "User-Agent": "handle-an-error-reply-lesson"},
    )
    try:
        with urllib.request.urlopen(request, timeout=10) as reply:
            status, body = reply.status, json.load(reply)
    except urllib.error.HTTPError as refusal:
        # A real API sends a refusal with a status of 400 or more, and urllib raises it as an error.
        # The error still carries the status and a body. This assumes the body is JSON.
        return refusal.code, json.load(refusal)
    # The practice API cannot send any status but 200. It puts the status a real server
    # would have sent inside the body, so that is the one we act on.
    if "error" in body:
        status = body["error"]["status"]
    return status, body


def decide(status, body):
    """What to do about one reply. Returns an action and, for a retry, how many seconds to wait."""
    if status < 400:
        return "use it", 0
    error = body["error"]
    if status == 429:
        return "wait and retry", error.get("retryAfterSeconds", 5)
    if status in (500, 502, 503, 504):
        return "wait and retry", 5        # the server's own trouble, which may pass
    if status == 401:
        return "sign in again", 0         # a new session is needed. Asking again will not make one
    return "stop and fix the request", 0  # 400, 403, 404, 422: the same request will be refused again


def call(path, wait=time.sleep):
    """Try a request up to MAX_ATTEMPTS times. Only an answer of wait and retry earns another try."""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        status, body = fetch(path)
        action, seconds = decide(status, body)
        if action == "use it":
            print("  attempt", attempt, "status", status, "worked")
            return action, attempt, body
        print("  attempt", attempt, "status", status, body["error"]["code"] + ":", action)
        if action != "wait and retry":
            return action, attempt, body
        if attempt < MAX_ATTEMPTS:
            wait(seconds)
    return "gave up", MAX_ATTEMPTS, body


def pretend_wait(seconds):
    # The practice file asks for 30 seconds and never changes its mind, so the lesson does not
    # sit through them. A real program would call time.sleep(seconds) here.
    print("    told to wait", seconds, "seconds. A real program would sleep here.")
    time.sleep(PAUSE)


results = {}
for path in ["/account.json", "/errors/session-expired.json", "/errors/rate-limited.json", "/errors/validation.json"]:
    print("GET", path)
    action, attempts, body = call(path, wait=pretend_wait)
    results[path] = (action, attempts)
    print("  outcome:", action, "after", attempts, "attempt" if attempts == 1 else "attempts")
    for field in body.get("error", {}).get("fields", []):
        print("    to correct:", field["field"], field["problem"] + ", received", repr(field["received"]))
    time.sleep(PAUSE)

# The check. Each kind of reply was handled in its own way, and nothing was retried without cause.
assert results["/account.json"] == ("use it", 1)
assert results["/errors/session-expired.json"] == ("sign in again", 1)
assert results["/errors/rate-limited.json"] == ("gave up", MAX_ATTEMPTS)
print("Check passed: one reply used, one stopped at once, one tried", MAX_ATTEMPTS, "times and no more.")

Your progress

Next: Idempotent orders

Your marks are kept in this browser only, in its local storage. They are not sent anywhere, there is no account, and they will not follow you to another browser or device. To remove them, use Clear my progress on the learning track page, or clear this site's data in your browser's settings. Marking needs JavaScript. With scripts off the track reads the same, without the ticks.

Where this sits in the path

Stage 4 of 4: Building things that survive. Failure, limits, repeated orders, logs, settings, tests and backtests that do not fool you: what separates a script from a tool.

  1. Retry and backoff
  2. Stay under a rate limit
  3. Handle an error replyYou are here
  4. Idempotent orders
  5. Log what happened
  6. Log your API calls
  7. Backtest an idea
  8. The look ahead mistake
  9. Settings from environment
  10. Test with saved replies
  11. A command line tool
  12. The capstone project

The whole path, every stage

Back: Stay under a rate limitNext: Idempotent orders