Write a client that pages through an API with cursor pagination and survives restarts.

How to answer

The loop is easy. The answer is one invariant, said early: “The saved cursor always points just past committed rows, and storing a page twice changes nothing.”

1. Pin the API’s contract. Is the cursor opaque, and how long does it live? What signals the end: a missing cursor, has_more, an empty page? Is the order stable? What are the rate limits?
2. Commit rows and cursor together. Write the page’s rows and the next cursor in one transaction. A crash before the commit replays that page; a crash after it resumes on the next.
3. Make the write idempotent. Upsert on the item’s id (SQLite and Postgres both have ON CONFLICT). Replayed pages become harmless, and so does an item seen on two pages because an update moved it in the sort order, or because the cursor is an offset underneath. If anything else writes these rows, update only when the record’s own updated_at is not older, so an older copy never replaces a newer one.
4. Retry only what can succeed. A GET is idempotent (RFC 9110, section 9.2.2), so it is safe to repeat: retry timeouts, 429 and the server errors that mean try again (500, 502, 503, 504) with jittered backoff, and honor Retry-After (RFC 9110, section 10.2.3). Stop on other 4xx errors. Asked to wait past your budget, exit: the saved cursor makes stopping cheap.
5. Guard the loop’s edges. End on a missing cursor, not an empty page. Fail loudly on a cursor you have already seen. On an expired cursor, resume from what you stored if the pages run in an order the API can filter on (in creation order, ask for created_after your newest stored record); rescan from the start only if it can’t. Upserts make both safe.
6. Say what changes when the sink is elsewhere. No shared transaction: the sink write must be idempotent itself, and the cursor saved only after the sink confirms.

The trap is saving progress and data as two independent steps in an unstated order.