Handle a failed call: retry, back off, give up

Calls fail. The network drops, the broker is busy, or you have asked too often. Code that assumes every call works will fail at the worst moment, so plan for failure from the start.

The first decision is which failures to retry. Being told to slow down is worth retrying after a pause. Being told your request is wrong is not: sending the same wrong request again only repeats the mistake, and with an order it could be dangerous.

The second is how long to wait. Doubling the wait after each refusal, called exponential backoff, gives a busy service room to recover. The third is when to stop. A loop that retries forever hides the problem instead of reporting it.

The example uses a pretend API that fails twice and then works, so you can run it anywhere. The error pages on this site list which real error codes fall into each group for each broker.

The code

# What to do when a call fails. Retry the right errors, wait longer each time, then give up.
# No broker and no internet: a pretend API fails twice, then works.
import time

attempts_seen = 0


class TooManyRequests(Exception):
    """Stands in for an HTTP 429 or a broker's allowance error."""


class BadRequest(Exception):
    """Stands in for an HTTP 400: the request itself is wrong, so retrying cannot help."""


def flaky_call():
    global attempts_seen
    attempts_seen += 1
    if attempts_seen <= 2:
        raise TooManyRequests("slow down")
    return {"status": "ok", "attempt": attempts_seen}


def call_with_backoff(call, max_attempts=5, first_wait=0.01):
    wait = first_wait
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except TooManyRequests:
            if attempt == max_attempts:
                raise                      # out of attempts: let the error through
            print("Attempt", attempt, "was refused. Waiting", wait, "seconds.")
            time.sleep(wait)
            wait = wait * 2                # double the wait each time
        # A BadRequest is not caught here on purpose. Retrying a wrong request just repeats the mistake.


print("Result:", call_with_backoff(flaky_call))


def always_wrong():
    raise BadRequest("missing a required field")


try:
    call_with_backoff(always_wrong)
except BadRequest as problem:
    print("Not retried:", problem)

What you should see

Attempt 1 was refused. Waiting 0.01 seconds.
Attempt 2 was refused. Waiting 0.02 seconds.
Result: {'status': 'ok', 'attempt': 3}
Not retried: missing a required field

Run with Python 3.12 on Windows. No network and no account.

IG limits, from the register

Errors you may meet

• error.security.account-access-deniedThe account has been denied login privileges.
• error.security.account-migratedThe account has been migrated to IG's client and account model.
• ACCOUNT_NOT_ENABLED_TO_TRADINGThe account is not enabled to trade. IG returns this in the reason field of a deal confirmation.
• error.security.account-not-yet-activatedThe account has not been activated yet.
• error.security.account-suspendedThe account has been suspended.
• error.security.account-token-invalidThe endpoint needs an account token and the one sent was not valid.

Check yourself

Answer each question from the code above before you open it. The answers were written from the script on this page and what it printed.

Exercise

Make the pretend API fail three times before it works, by changing one number in flaky_call. Predict the new output first.

Attempt 1 was refused. Waiting 0.01 seconds.
Attempt 2 was refused. Waiting 0.02 seconds.
Attempt 3 was refused. Waiting 0.04 seconds.
Result: {'status': 'ok', 'attempt': 4}
Not retried: missing a required field

# Exercise solution: the lesson script with the one change the exercise asks for.
# What to do when a call fails. Retry the right errors, wait longer each time, then give up.
# No broker and no internet: a pretend API fails twice, then works.
import time

attempts_seen = 0


class TooManyRequests(Exception):
    """Stands in for an HTTP 429 or a broker's allowance error."""


class BadRequest(Exception):
    """Stands in for an HTTP 400: the request itself is wrong, so retrying cannot help."""


def flaky_call():
    global attempts_seen
    attempts_seen += 1
    if attempts_seen <= 3:
        raise TooManyRequests("slow down")
    return {"status": "ok", "attempt": attempts_seen}


def call_with_backoff(call, max_attempts=5, first_wait=0.01):
    wait = first_wait
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except TooManyRequests:
            if attempt == max_attempts:
                raise                      # out of attempts: let the error through
            print("Attempt", attempt, "was refused. Waiting", wait, "seconds.")
            time.sleep(wait)
            wait = wait * 2                # double the wait each time
        # A BadRequest is not caught here on purpose. Retrying a wrong request just repeats the mistake.


print("Result:", call_with_backoff(flaky_call))


def always_wrong():
    raise BadRequest("missing a required field")


try:
    call_with_backoff(always_wrong)
except BadRequest as problem:
    print("Not retried:", problem)

Your progress

Next: Stay under a rate limit

Your marks are kept in this browser only, in its local storage. They are not sent anywhere, there is no account, and they will not follow you to another browser or device. To remove them, use Clear my progress on the learning track page, or clear this site's data in your browser's settings. Marking needs JavaScript. With scripts off the track reads the same, without the ticks.

This browser refused to store anything for this page, so marks cannot be kept here.

Where this sits in the path

Stage 4 of 4: Building things that survive. Failure, limits, repeated orders, logs, settings, tests and backtests that do not fool you: what separates a script from a tool.

1. 21Retry and backoffYou are here
2. 22Stay under a rate limit
3. 23Handle an error reply
4. 24Idempotent orders
5. 25Log what happened
6. 26Log your API calls
7. 27Backtest an idea
8. 28The look ahead mistake
9. 29Settings from environment
10. 30Test with saved replies
11. 31A command line tool
12. EndThe capstone project

The whole path, every stage