The Sparrow Dataset Format

Open specification, v1.0-draft · 2026-08-07

A Sparrow dataset is a folder of self-describing time-series files that any client can serve as live, formula-addressable data. It is deliberately boring: columnar files, a naming rule, a metadata sidecar, and one honesty rule about timestamps. Anything that can host static files can publish it; anything that can read Parquet can consume it.

The same folder works over every transport:

file:C:\datasets          local disk
file:\\nas\share          SMB / network share
https://host/path         any static web host or CDN
s3://bucket/path          any S3-compatible object store

The words MUST / SHOULD / MAY are used in the RFC-2119 sense.

1 · The folder

A dataset folder contains one data file per dataset prefix, plus optional underscore-prefixed sidecars:

PET.vortex           data for all series whose ID starts "PET."
ELEC.parquet         data for all series whose ID starts "ELEC."
ACME.vortex          a third publisher's dataset — coexists freely
_META.parquet        series catalog (names, units, frequencies)   [optional]
_INDEX.json          file listing for HTTP hosting                [optional]

Routing rule: a series ID's first dot-segment names its file. PET.RWTC.D → PET.vortex (or PET.parquet if no .vortex exists). Readers MUST try .vortex first, then .parquet.
File names beginning with _ are reserved for dataset-level artifacts.
Adding a dataset is copying one file into the folder. No registration, no restart, no configuration. Removing it is deleting the file.

2 · Data files

Each data file holds exactly three columns:

column	type	rules
`series_id`	string	UTF-8; dot-separated hierarchy; first segment MUST equal the file's prefix; no spaces recommended
`period`	date (date32 recommended)	see period semantics below
`value`	float64	nulls permitted; readers MAY drop them

Rows SHOULD be sorted by (series_id, period) ascending — readers work regardless, but sorted files let columnar readers prune to a single series in milliseconds on tens of millions of rows.
Formats: Vortex (fast scans, predicate pushdown) or Parquet (universal). One format per file; a folder MAY mix formats across files.
Period semantics: period is always a calendar date. Coarser granularities use the FIRST day of the period: monthly → the 1st, quarterly → the first day of the quarter, semi-annual → Jan 1 / Jul 1, annual → Jan 1. Parquet string periods ("2024-01-02", "20240102") are tolerated; dates are canonical.
Frequency is conveyed by the series-ID suffix convention — .D .W .M .Q .A — and/or the catalog's freq column. Readers MAY infer it from median period gaps when absent.

3 · _META.parquet — the catalog

One row per series makes the folder self-describing: search, names and units work with no server anywhere.

column	type	notes
`series_id`	string	unique key
`name`	string	human-readable; empty allowed
`units`	string	e.g. `Dollars per Barrel`; empty allowed
`freq`	string	`D W M Q A` or empty

Rows SHOULD be sorted by series_id.
Multi-publisher rule: a publisher updating the catalog MUST preserve rows whose prefix it does not own (upsert by prefix, never truncate). Two publishers sharing a folder must never erase each other.

4 · _INDEX.json — HTTP discovery

Static HTTP has no directory listing, so hosted datasets SHOULD publish:

{
  "sparrow_spec": "1.0",
  "files": ["PET.vortex", "ACME.vortex", "_META.parquet", "_INDEX.json"],
  "generated": "2026-08-07"
}

Only files is required. Extra keys (sizes, hashes, timestamps) are permitted and ignored by readers that don't understand them. Readers MUST fall back to probing PREFIX.vortex / PREFIX.parquet when the index is absent, and SHOULD remember misses.

5 · The honesty rule — freshness

This is the load-bearing rule that makes "NEW DATA" signals trustworthy:

Publishers MUST replace files atomically (write temp + rename, or an atomic object PUT). Readers must never observe a partial file.
Publishers SHOULD skip the write entirely when content is unchanged (compare a content hash against the previous publish), so that:
A file's modification time moves if and only if its data changed. Over HTTP/S3 the same rule applies to Last-Modified and ETag.

A client may then treat mtime / Last-Modified as the freshness signal — cheap to poll (one stat or HEAD per file), impossible to cry wolf.

6 · Transport notes

transport	discovery	freshness	fetch
`file:` / UNC	directory listing	file mtime	direct read (UNC SHOULD be cached locally, copy-if-newer)
`https://`	`_INDEX.json` (else probe)	`HEAD` → `Last-Modified`/`ETag`	conditional `GET`, cached per source
`s3://`	same as https	same as https	public or presigned URLs; SigV4 signing is a client option, not part of this spec

Whole files are transferred (then cached); this format trades query pushdown for zero infrastructure. Clients wanting server-side slicing or computation should graduate to an Arrow Flight endpoint — the formulas don't change.

7 · Compatibility & versioning

Unknown files in the folder MUST be ignored.
Unknown columns in _META.parquet and unknown keys in _INDEX.json MUST be ignored.
This document is 1.0-draft. Breaking changes bump the major version and announce themselves via sparrow_spec in _INDEX.json.

8 · Reference implementations

Publisher: make_dataset.py — any CSV → conforming dataset files + catalog upsert, ~80 lines of Python. A production pipeline publishes 6 datasets + catalog daily with the content-hash rule.
Consumer: Sparrow for Excel — formulas, tables, freshness signals, revision tracking, offline analytics over any folder conforming to this spec.

A minimal valid dataset is one Parquet file with three columns and a conforming name. Everything else is optional. That's the point.