Scale
Counted across raw observations from grid, market and fundamentals feeds; every satellite product we access, read pixel by pixel and band by band; and the values derived from each of them along the way.
Object storage acts as the database for the entire system. There is no relational database, cache or message queue: every serverless function reads its inputs from storage and writes its output back, from raw API pulls to model artifacts, signal records and trade state. Nothing reaches a model, or a decision, without passing through it.
What comes in every day
Grid data, every five minutes. Prices and generation mix are pulled from the Independent System Operators that run the U.S. grid. A single five-minute pricing feed adds 288 observations per day, per data type, per ISO.
Satellite imagery, every pass. Thermal, radar and optical imagery is read pixel by pixel and band by band, across 25 refinery and storage sites on the Gulf Coast. A single radar scene alone carries hundreds of millions of measurements.
Model outputs, several times a day. The unsupervised models run three times daily, the grid-stress indicator runs daily, the spread models run every weekday, and agent88 runs its entropy scans every day. Each run writes a signal record back to storage.
Decisions and outcomes, every trading day. Analyst theses, execution decisions and exit logs are logged as they happen, and an end-of-day P&L snapshot is written after the close.
Weekly. EIA storage reports arrive, the sequence models retrain on everything accumulated so far, and the past week’s trades are reviewed against what actually happened.
One reading becomes many
No observation is used once. Every new reading is compared with the ones before it, and each comparison is a new data point: the change since the last reading, the 1-day and 5-day change, its rolling 30-day z-score, the probability of each grid regime, the autoencoder’s reconstruction error, and the gap to modeled fair value.
Those derived values are compared again on the next run. A single five-minute price feeds changes, z-scores and regime probabilities, and every satellite pixel becomes part of a per-site trend, an anomaly score and a regime label. That is how a few raw feeds turn into billions, then trillions, of points behind each decision.
Files over time
Files in the main datalake, weekly. Other data stores and imagery read directly from satellite archives are not included. Source: object storage metrics.
The latest outputs are overwritten on each run, but a date-stamped archive is kept alongside them, so the history only gets longer.
Why it compounds
More history makes every model sharper. The satellite and grid models retrain on everything accumulated so far. Each retraining cycle ingests all accumulated site-days and re-fits the autoencoder, updating anomaly thresholds from the new error distribution. Early on, with little history, thresholds come from a historical distribution; as months of real-time data accumulate, the percentile thresholds become more accurate, the false-positive rate drops, and rare, genuine anomalies stand out more clearly.
The same record feeds the reasoning layer. Analyst theses, execution decisions and exit logs are written back to the lake, and reviewed weekly against what actually happened.
For how the stack keeps its energy footprint small, see Environmental Considerations & Conclusion in Paper #2.