grain-bids-to-excel

One person, one tedious job

Grain elevators publish "cash bids" (what they will pay today for corn, soybeans, wheat and so on) on their public websites. Every site does it differently. Some are plain HTML tables. Some are third-party widgets that only render after JavaScript runs. Some are lists of <div>s pretending to be a table.

Column names disagree. Prices show up as 4.23, as 423-0s, or as a basis you have to add to a futures price before the number means anything. Someone whose job depended on that data was visiting each site by hand, reading the tables, and retyping the numbers into a spreadsheet. Every day.

This app does that instead. Add companies once, click "Run a scan", get a formatted .xlsx workbook. It is not a platform. It is a small sharp tool built for one non-technical user, and it removed a daily chore.

A local Flask app that serves only on 127.0.0.1. The browser is just the window; nothing is exposed to the network. No login, no cloud, no LLM anywhere in the extraction path.

Three engineering decisions worth reading

Zip it and email it

The user could not be asked to install a toolchain. So the deliverable is a folder: double-click SETUP once, double-click RUN every time, Windows and macOS. A vendored embeddable interpreter with an edited ._pth, a mac branch that picks its build by uname -m, and a packaging check that requires import flask to fail in the bundled interpreter before shipping, so a dev run cannot contaminate it.

Scoring, not selectors

A CSS selector per elevator does not survive the first redesign. The extractor discovers candidate containers (including virtual grids built by grouping sibling elements on tag plus class signature), then scores each on eight independent signals. Above 8.0 it is accepted; between 4.0 and 8.0 the rows still come out, marked low_confidence instead of silently trusted or silently dropped.

Domain arithmetic as a parser

Bid tables often carry two signed-decimal columns a parser cannot tell apart: basis and day change. The extractor tests each candidate against the trade identity basis == cash - futures within 0.02 and takes the column that agrees on at least 70% of rows. It also decodes the trade's eighths notation: 415-0s is $4.15, +0-2 is +$0.0025.

How it fits together

A browser is a last resort. Tier 1 is requests; escalation to Playwright Chromium happens on three specific conditions, not by default. Failures never raise to the UI: every one becomes a flag with plain English written for a grain merchandiser, so one bad site degrades a run to partial instead of aborting it.

SETUP / RUN launchers        double-clickable, no admin rights, no system Python
        |
        v
launcher.py                  lock file + /api/ping handshake, waitress on 127.0.0.1
        |
        v
web/         6 blueprints, 35 routes: 6 rendered pages, 29 JSON endpoints
        |    Jinja + vanilla CSS/JS, no build step, no node_modules
        v
services/    ScanManager: one thread, one queue, 2000-event ring buffer
        |    APScheduler enqueues through the same single lane
        v
engine/      fetcher -> detect -> extractor -> headers -> normalize
        |    discover / resolve / probe / staleness / store / pipeline
        v
report/      builder -> queries -> health -> layout -> charts -> naming -> files
        |
        v
Spreadsheets/*.xlsx          master / per-company / per-commodity workbooks

One module touches SQLite. Eleven tables, WAL mode, single-writer discipline, versioned forward-only migrations.

Preferences in JSON, records in SQLite, and the two never mix. The whole settings document is schema-validated on every write, with per-company overrides.

Excel output is hardened for someone who will not debug it. Atomic writes, never overwrite, a fallback name when the file is open in Excel, and a test that unzips the workbook to prove there are no external links.

Nothing installs outside the folder. One paths.py defines the entire on-disk contract, and file endpoints refuse any path outside the Spreadsheets directory.

It can run itself at 7:30 every morning when the schedule is turned on.

Quickstart

For the person it was built for, the entire procedure is three steps.

Unzip the folder anywhere.

Double-click SETUP. Once. Needs internet, no admin rights.

Double-click RUN. The browser opens by itself.

For a developer, the public repository does not ship the bundled interpreter or the browsers, so run it against your own Python.

py -3 -m pip install -r app/requirements.txt pytest
py -3 -m playwright install chromium    # only for Tier-2 JS pages

py -3 -m pytest app/tests               # 107 passed
py -3 app/src/launcher.py               # http://127.0.0.1:8383

Honest notes

Extraction is heuristic. A low-scoring page still returns rows, flagged low_confidence, so output is worth spot-checking when a new site is added.

Search-engine name resolution is fragile by nature. Three fallback transports exist because search engines block scripted clients, and the user always confirms a match rather than the app auto-accepting one.

robots.txt is honored by default, with a 24-hour cache and a 2 to 5 second jittered politeness delay per domain. The check is deliberately fail-open. The toggle exists under Advanced settings and turning it off is the operator's call.

Single-user by design. No auth, no concurrent writers, no deployment story beyond 127.0.0.1.

Tests run in CI on every push and pull request, and live scans were verified by hand.

Working and shipped. It was used for its actual purpose. It is not under active development.