Stage 3 of 4, lesson 19 of 31. Improver

Pagination over HTTP: follow the pointer to the next page

Collect a trade history from the practice API one page at a time, by following the pointer each page gives to the next, with a pause between requests and a proof that nothing is missing.

Independent and unofficialCode run 12 Oct 2026

Risk warning: CFDs and spread bets are complex instruments and come with a high risk of losing money rapidly due to leverage. You can lose money on these products, and each provider publishes its own loss figure under FCA rules. You should consider whether you understand how they work and whether you can afford to take the high risk of losing your money. Automated and API trading can amplify losses through bugs, connectivity failures and runaway orders.

You need: The lessons on paged results and on your first HTTP request. An internet connection. No broker account. No key.

The last lesson collected pages from a file by counting: page one, page two, until the total. Many APIs do not let you count. Each reply carries a pointer to the page after it, and the last reply carries none. Your loop follows the pointer until there is nothing to follow.

The practice API serves twelve invented closed trades on three pages, newest first. Each page has a paging object. Its next field holds the path of the following page, and on the last page it is null, which Python reads as None.

A loop that follows pointers trusts the server to end it, so the script adds two guards of its own. It keeps a list of the pages it has read and stops if a pointer leads back to one of them. And it has a limit on the number of pages, so that a mistake costs twenty requests and not twenty thousand.

It pauses between pages, but not before the first. On a real API each page is a request that counts against your limit, and a long history can use a minute's allowance in a second.

The check at the end is what makes the result worth having. The reply says how many rows exist. The script refuses to carry on unless it collected exactly that many, and each of them once. An export that is silently short looks just like a complete one until the totals are wrong.

The practice API this lesson calls

Practice data, not any broker's API: fixed files of invented values on this site. Each opens in a browser tab, so you can read what the script will be sent.

Every address on the practice API, and what it cannot do

The code

# Collect a trade history that arrives in pages over HTTP, by following each page's pointer to the next.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key, and it cannot place a trade.
import json
import os
import time
import urllib.request
from decimal import Decimal

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1        # seconds to leave between pages
MAX_PAGES = 20   # a loop that follows pointers needs a limit it cannot talk its way past


def get(path):
    request = urllib.request.Request(
        BASE_URL + path,
        headers={"Accept": "application/json", "User-Agent": "follow-the-next-page-lesson"},
    )
    with urllib.request.urlopen(request, timeout=10) as reply:
        data = json.load(reply, parse_float=Decimal)
    assert data["practice"] is True, "this reply is not practice data"
    return data


trades = []
visited = []
path = "/trades/page-1.json"
while path is not None:
    if path in visited:
        raise RuntimeError("the pointer led back to a page already read: " + path)
    if len(visited) == MAX_PAGES:
        raise RuntimeError("gave up after " + str(MAX_PAGES) + " pages")
    if visited:
        time.sleep(PAUSE)          # not before the first page, only between pages
    answer = get(path)
    visited.append(path)
    paging = answer["paging"]
    trades.extend(answer["trades"])
    print("Page", paging["page"], "of", paging["totalPages"], "gave", len(answer["trades"]), "trades. Next:", paging["next"])
    path = paging["next"]          # None on the last page, which ends the loop

# The check. Every row the API says it has was collected, and none of them twice.
ids = [trade["tradeId"] for trade in trades]
assert len(set(ids)) == len(ids), "the same trade arrived twice"
assert len(trades) == paging["totalRows"], "rows are missing"

print("Requests made:", len(visited))
print("Trades collected:", len(trades))
print("Newest:", trades[0]["tradeId"], trades[0]["closedAt"])
print("Oldest:", trades[-1]["tradeId"], trades[-1]["closedAt"])
print("Total profit or loss:", sum(trade["profitLoss"] for trade in trades), "GBP")
print("Check passed: collected every row the API said it had, each one once.")

What you should see

Page 1 of 3 gave 5 trades. Next: /trades/page-2.json
Page 2 of 3 gave 5 trades. Next: /trades/page-3.json
Page 3 of 3 gave 2 trades. Next: None
Requests made: 3
Trades collected: 12
Newest: T-3012 2026-01-01T10:15:00Z
Oldest: T-3001 2025-12-19T15:40:00Z
Total profit or loss: 42.4 GBP
Check passed: collected every row the API said it had, each one once.

Run with Python 3.12 on Windows against a copy of the practice API files served over HTTP on the same machine, with PRACTICE_API set to the address of the copy. They are the files this site serves. No account and no key.

Check yourself

Answer each question from the code above before you open it. The answers were written from the script on this page and what it printed.

What makes the loop stop?

On page 3 the next field is null, which json.load turns into None. The script puts it in path, and while path is not None is then false.

Which two lines would stop the loop if the API never sent a null?

The two that raise RuntimeError. One fires if a pointer names a page already in visited. The other fires when the number of pages read reaches MAX_PAGES.

Three requests were made. How many pauses were there, and why?

Two. time.sleep runs only if visited already holds a page, so there is no wait before the first request and one before each of the others.

Exercise

The pages arrive newest first. Change the script so that it keeps only trades closed on or after 24 December 2025, and stops asking for pages as soon as a page ends with an older trade. It should make two requests, not three. The check on the number of rows no longer applies, so replace it with one that suits.

What the changed script prints

Page 1 of 3 gave 5 trades. Next: /trades/page-2.json
Page 2 of 3 gave 5 trades. Next: /trades/page-3.json
This page ends with a trade older than 2025-12-24, so no more pages are needed.
Requests made: 2
Trades collected: 7
Newest: T-3012 2026-01-01T10:15:00Z
Oldest: T-3006 2025-12-24T09:10:00Z
Total profit or loss: 17.7 GBP
Check passed: kept only trades closed on or after 2025-12-24, each one once.

Captured from a run of the solution on 12 Oct 2026. Run with Python 3.12 on Windows against a copy of the practice API files served over HTTP on the same machine, with PRACTICE_API set to the address of the copy. They are the files this site serves. No account and no key.

One solution

# Exercise solution: the lesson script with the one change the exercise asks for.
# Collect a trade history that arrives in pages over HTTP, by following each page's pointer to the next.
# The practice API is a set of fixed files of invented values. It is not any broker's API.
# It needs no account and no key, and it cannot place a trade.
import json
import os
import time
import urllib.request
from decimal import Decimal

BASE_URL = os.environ.get("PRACTICE_API", "https://openbrokerapi.com/mock/v1")
PAUSE = 1        # seconds to leave between pages
MAX_PAGES = 20   # a loop that follows pointers needs a limit it cannot talk its way past
SINCE = "2025-12-24"   # keep trades closed on or after this date


def get(path):
    request = urllib.request.Request(
        BASE_URL + path,
        headers={"Accept": "application/json", "User-Agent": "follow-the-next-page-lesson"},
    )
    with urllib.request.urlopen(request, timeout=10) as reply:
        data = json.load(reply, parse_float=Decimal)
    assert data["practice"] is True, "this reply is not practice data"
    return data


trades = []
visited = []
path = "/trades/page-1.json"
while path is not None:
    if path in visited:
        raise RuntimeError("the pointer led back to a page already read: " + path)
    if len(visited) == MAX_PAGES:
        raise RuntimeError("gave up after " + str(MAX_PAGES) + " pages")
    if visited:
        time.sleep(PAUSE)          # not before the first page, only between pages
    answer = get(path)
    visited.append(path)
    paging = answer["paging"]
    trades.extend(trade for trade in answer["trades"] if trade["closedAt"] >= SINCE)
    print("Page", paging["page"], "of", paging["totalPages"], "gave", len(answer["trades"]), "trades. Next:", paging["next"])
    path = paging["next"]          # None on the last page, which ends the loop
    if answer["trades"][-1]["closedAt"] < SINCE:
        print("This page ends with a trade older than", SINCE + ", so no more pages are needed.")
        path = None

# The check. Every row the API says it has was collected, and none of them twice.
ids = [trade["tradeId"] for trade in trades]
assert len(set(ids)) == len(ids), "the same trade arrived twice"
assert all(trade["closedAt"] >= SINCE for trade in trades), "an older trade was kept"

print("Requests made:", len(visited))
print("Trades collected:", len(trades))
print("Newest:", trades[0]["tradeId"], trades[0]["closedAt"])
print("Oldest:", trades[-1]["tradeId"], trades[-1]["closedAt"])
print("Total profit or loss:", sum(trade["profitLoss"] for trade in trades), "GBP")
print("Check passed: kept only trades closed on or after", SINCE + ", each one once.")

Your progress

Next: Price history to CSV

Your marks are kept in this browser only, in its local storage. They are not sent anywhere, there is no account, and they will not follow you to another browser or device. To remove them, use Clear my progress on the learning track page, or clear this site's data in your browser's settings. Marking needs JavaScript. With scripts off the track reads the same, without the ticks.

Where this sits in the path

Stage 3 of 4: Talking to a broker. What an API sends back, real requests to a practice API that needs no account, how to keep a key safe, a first read only call on a demo account, and histories that arrive in pages.

The whole path, every stage

Back: Paged results Next: Price history to CSV