Write tests with unittest, on saved replies, with no network

Plain asserts took you a long way. unittest, which comes with Python, adds what a growing program needs: tests grouped in a class, a name for each, helpers such as assertEqual and assertRaises that say what differed, and a count at the end. Each method whose name starts with test_ is one test.

Code that talks to an API is awkward to test, because a test that needs the network is slow, fails when you are offline, and uses your allowance. The answer is to save a real reply once and test against the copy. The script does it in two steps. save_replies fetches two files and writes them into a folder. The tests then read only that folder.

setUp runs before every test. Here it swaps urllib's urlopen for a function that fails, so a test that tried to make a request would stop and say so. That turns no network from a hope into something the tests prove.

The replies are order confirmations: what a program reads back after sending an order, to learn what became of it. One was accepted and one was rejected. They are fixed files about orders that were never sent, and reading them changes nothing. The function under test turns each into a line a person can read, and refuses a status it does not know, because a program should never guess what became of an order.

A saved reply can also be bent. One test loads the accepted confirmation and changes its status to a value the API has never sent you, to prove the function refuses it. You cannot ask a server to misbehave on demand. A copy on disk will do whatever you need.

Outside a lesson you would keep tests in a file of their own, with a name that starts with test_, and run them with python -m unittest. This script runs them itself and prints a line for each, so that the lesson is one file.

The practice API this lesson calls

Practice data, not any broker's API: fixed files of invented values on this site. Each opens in a browser tab, so you can read what the script will be sent.

• /mock/v1/confirms/my-order-0001.json
• /mock/v1/confirms/my-order-0002.json

Every address on the practice API, and what it cannot do

The code

# Test your own functions with unittest, on replies saved to disk, so the tests need no network.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key. The confirmations describe orders that were never sent.
import json
import os
import time
import unittest
import urllib.request
from unittest import mock

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1   # seconds to leave between calls
FOLDER = "saved-replies"
SAVED = {"accepted.json": "/confirms/my-order-0001.json", "rejected.json": "/confirms/my-order-0002.json"}


def read_confirmation(reply):
    """What became of an order, from the confirmation an API sends back. This is the function under test."""
    if reply.get("practice") is not True:
        raise ValueError("this reply is not practice data")
    confirmation = reply["confirmation"]
    if confirmation["status"] == "ACCEPTED":
        return "Order " + confirmation["orderId"] + " accepted: " + confirmation["direction"] + " " + str(confirmation["size"]) + " " + confirmation["instrumentId"]
    if confirmation["status"] == "REJECTED":
        return "Order rejected: " + confirmation["reason"] + ". Nothing was placed"
    raise ValueError("a status this program does not know: " + confirmation["status"])


def save_replies():
    """Step one, done once, with the network: fetch each reply and keep it as a file."""
    os.makedirs(FOLDER, exist_ok=True)
    for number, (name, path) in enumerate(SAVED.items()):
        if number:
            time.sleep(PAUSE)
        request = urllib.request.Request(BASE_URL + path, headers={"Accept": "application/json", "User-Agent": "saved-replies-lesson"})
        with urllib.request.urlopen(request, timeout=10) as reply:
            text = reply.read().decode("utf-8")
        with open(os.path.join(FOLDER, name), "w", encoding="utf-8") as f:
            f.write(text)
        print("Saved", path, "as", FOLDER + "/" + name)


def saved(name):
    with open(os.path.join(FOLDER, name), encoding="utf-8") as f:
        return json.load(f)


class ConfirmationTests(unittest.TestCase):
    """Step two, done as often as you like, with no network: every test reads a saved reply."""

    def setUp(self):
        # Before each test, swap the function that opens addresses for one that fails.
        # A test that tried to use the network would stop here and say so.
        blocker = mock.patch("urllib.request.urlopen", side_effect=AssertionError("a test tried to use the network"))
        blocker.start()
        self.addCleanup(blocker.stop)

    def test_accepted_order_names_its_id(self):
        self.assertEqual(read_confirmation(saved("accepted.json")), "Order O-2001 accepted: BUY 3 TIDE")

    def test_rejected_order_gives_the_reason(self):
        self.assertIn("INSUFFICIENT_FUNDS", read_confirmation(saved("rejected.json")))

    def test_unknown_status_is_refused(self):
        reply = saved("accepted.json")
        reply["confirmation"]["status"] = "PENDING"      # a saved reply can be bent into a case you have not met
        with self.assertRaises(ValueError):
            read_confirmation(reply)

    def test_reply_that_is_not_practice_data_is_refused(self):
        with self.assertRaises(ValueError):
            read_confirmation({"confirmation": {"status": "ACCEPTED"}})


save_replies()

suite = unittest.defaultTestLoader.loadTestsFromTestCase(ConfirmationTests)
names = [test.id().split(".")[-1] for test in suite]
result = unittest.TestResult()
suite.run(result)
failed = [test.id().split(".")[-1] for test, why in result.failures + result.errors]
for name in names:
    print("FAIL" if name in failed else "pass", name)
print(result.testsRun - len(failed), "of", result.testsRun, "tests passed, with the network switched off for each one")
assert result.wasSuccessful(), "a test failed"

# The function at work on the two saved replies.
print(read_confirmation(saved("accepted.json")))
print(read_confirmation(saved("rejected.json")))

What you should see

Saved /confirms/my-order-0001.json as saved-replies/accepted.json
Saved /confirms/my-order-0002.json as saved-replies/rejected.json
pass test_accepted_order_names_its_id
pass test_rejected_order_gives_the_reason
pass test_reply_that_is_not_practice_data_is_refused
pass test_unknown_status_is_refused
4 of 4 tests passed, with the network switched off for each one
Order O-2001 accepted: BUY 3 TIDE
Order rejected: INSUFFICIENT_FUNDS. Nothing was placed

Run with Python 3.12 on Windows against a copy of the practice API files served over HTTP on the same machine, with PRACTICE_API set to the address of the copy. They are the files this site serves. No account and no key.

Check yourself

Answer each question from the code above before you open it. The answers were written from the script on this page and what it printed.

Exercise

Make read_confirmation end the line for an accepted order with the level, so that it reads at 61.5. Run the script before you touch the tests and see which one fails. Then change that test to expect the new line.

Saved /confirms/my-order-0001.json as saved-replies/accepted.json
Saved /confirms/my-order-0002.json as saved-replies/rejected.json
pass test_accepted_order_names_its_id
pass test_rejected_order_gives_the_reason
pass test_reply_that_is_not_practice_data_is_refused
pass test_unknown_status_is_refused
4 of 4 tests passed, with the network switched off for each one
Order O-2001 accepted: BUY 3 TIDE at 61.5
Order rejected: INSUFFICIENT_FUNDS. Nothing was placed

# Exercise solution: the lesson script with the one change the exercise asks for.
# Test your own functions with unittest, on replies saved to disk, so the tests need no network.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key. The confirmations describe orders that were never sent.
import json
import os
import time
import unittest
import urllib.request
from unittest import mock

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1   # seconds to leave between calls
FOLDER = "saved-replies"
SAVED = {"accepted.json": "/confirms/my-order-0001.json", "rejected.json": "/confirms/my-order-0002.json"}


def read_confirmation(reply):
    """What became of an order, from the confirmation an API sends back. This is the function under test."""
    if reply.get("practice") is not True:
        raise ValueError("this reply is not practice data")
    confirmation = reply["confirmation"]
    if confirmation["status"] == "ACCEPTED":
        return "Order " + confirmation["orderId"] + " accepted: " + confirmation["direction"] + " " + str(confirmation["size"]) + " " + confirmation["instrumentId"] + " at " + str(confirmation["level"])
    if confirmation["status"] == "REJECTED":
        return "Order rejected: " + confirmation["reason"] + ". Nothing was placed"
    raise ValueError("a status this program does not know: " + confirmation["status"])


def save_replies():
    """Step one, done once, with the network: fetch each reply and keep it as a file."""
    os.makedirs(FOLDER, exist_ok=True)
    for number, (name, path) in enumerate(SAVED.items()):
        if number:
            time.sleep(PAUSE)
        request = urllib.request.Request(BASE_URL + path, headers={"Accept": "application/json", "User-Agent": "saved-replies-lesson"})
        with urllib.request.urlopen(request, timeout=10) as reply:
            text = reply.read().decode("utf-8")
        with open(os.path.join(FOLDER, name), "w", encoding="utf-8") as f:
            f.write(text)
        print("Saved", path, "as", FOLDER + "/" + name)


def saved(name):
    with open(os.path.join(FOLDER, name), encoding="utf-8") as f:
        return json.load(f)


class ConfirmationTests(unittest.TestCase):
    """Step two, done as often as you like, with no network: every test reads a saved reply."""

    def setUp(self):
        # Before each test, swap the function that opens addresses for one that fails.
        # A test that tried to use the network would stop here and say so.
        blocker = mock.patch("urllib.request.urlopen", side_effect=AssertionError("a test tried to use the network"))
        blocker.start()
        self.addCleanup(blocker.stop)

    def test_accepted_order_names_its_id(self):
        self.assertEqual(read_confirmation(saved("accepted.json")), "Order O-2001 accepted: BUY 3 TIDE at 61.5")

    def test_rejected_order_gives_the_reason(self):
        self.assertIn("INSUFFICIENT_FUNDS", read_confirmation(saved("rejected.json")))

    def test_unknown_status_is_refused(self):
        reply = saved("accepted.json")
        reply["confirmation"]["status"] = "PENDING"      # a saved reply can be bent into a case you have not met
        with self.assertRaises(ValueError):
            read_confirmation(reply)

    def test_reply_that_is_not_practice_data_is_refused(self):
        with self.assertRaises(ValueError):
            read_confirmation({"confirmation": {"status": "ACCEPTED"}})


save_replies()

suite = unittest.defaultTestLoader.loadTestsFromTestCase(ConfirmationTests)
names = [test.id().split(".")[-1] for test in suite]
result = unittest.TestResult()
suite.run(result)
failed = [test.id().split(".")[-1] for test, why in result.failures + result.errors]
for name in names:
    print("FAIL" if name in failed else "pass", name)
print(result.testsRun - len(failed), "of", result.testsRun, "tests passed, with the network switched off for each one")
assert result.wasSuccessful(), "a test failed"

# The function at work on the two saved replies.
print(read_confirmation(saved("accepted.json")))
print(read_confirmation(saved("rejected.json")))

Your progress

Next: A command line tool

Your marks are kept in this browser only, in its local storage. They are not sent anywhere, there is no account, and they will not follow you to another browser or device. To remove them, use Clear my progress on the learning track page, or clear this site's data in your browser's settings. Marking needs JavaScript. With scripts off the track reads the same, without the ticks.

This browser refused to store anything for this page, so marks cannot be kept here.

Where this sits in the path

Stage 4 of 4: Building things that survive. Failure, limits, repeated orders, logs, settings, tests and backtests that do not fool you: what separates a script from a tool.

1. 21Retry and backoff
2. 22Stay under a rate limit
3. 23Handle an error reply
4. 24Idempotent orders
5. 25Log what happened
6. 26Log your API calls
7. 27Backtest an idea
8. 28The look ahead mistake
9. 29Settings from environment
10. 30Test with saved repliesYou are here
11. 31A command line tool
12. EndThe capstone project

The whole path, every stage