# Data

Version 0.1, 2026-10-12. Source: https://developer.nrgon.net/data

The second layer: how data is stored and where each kind goes. Everything above reads from here.

## Formats

| format | used for | rule |
|---|---|---|
| JSON | one message, one record, one API response | UTF-8, one object per file, keys in `snake_case`, units in the key name |
| JSONL | many records of the same kind in one file: a day of telemetry, an event log | one JSON object per line, append only, no header |
| CSV | tables people open in a spreadsheet: interval load, site lists, catalogues | UTF-8, comma, header row, ISO 8601 dates, no thousands separators |
| YAML | configuration and definitions a person edits: sites, assets, value cases, data requests | in git, comments allowed, one document per file |
| Markdown | text: READMEs, reports, notes | plain, tables for facts |
| PDF | documents as received and reports as delivered | kept unchanged; values extracted into JSON next to it |
| XLSX, images, drawings | as received from partners and sites | kept unchanged in the bucket or the Drive folder |

Time is ISO 8601 in UTC, `2026-09-24T10:04:43Z`. Numbers carry the unit in the field
name: `power_kw`, `energy_kwh`, `soc_pct`, `temp_c`.

## Where data goes

```schematic
col Kind | live/live/Live data | dump/dump/One-time dump | doc/doc/Documents | photo/photo/Site photos | config/config/Configuration | report/report/Reports
col Destination | redis/redis/Redis, latest | bucket/bucket/Bucket | drive/drive/Drive folder | git/git/Git
edges: live>redis live>bucket dump>bucket doc>drive doc>bucket photo>drive config>git report>bucket report>drive
```

| kind | example | goes to | as |
|---|---|---|---|
| live data | battery state every minute, a swap event, an alarm | bucket, one object per message; Redis, latest value per asset | JSON per message; JSONL per day for analysis |
| one-time dump | 12 months of interval load from a utility, an asset list from a partner | bucket, under `<source>/dump/<date>/`, as delivered, with a `manifest.json` naming each file and its origin | CSV, XLSX, ZIP, as received |
| documents | bills, specifications, single-line diagrams, contracts | Drive folder of the project, so the copy people edit is the copy used; extracted values to the bucket | PDF, DOCX, XLSX |
| site photos and notes | installation photos, meter readings, handover notes from on-site teams | Drive folder, one subfolder per site and visit. Taken on a phone, uploaded from the Drive app | JPG, HEIC, Markdown |
| configuration | sites, assets, value cases, tariffs to apply, data request lists | git, next to the component that reads it | YAML |
| reports | proposal, weekly report, final report | bucket under `<project>/reports/`; a copy in the Drive folder for the partner | PDF and Markdown |

On-site operators work in the Drive folder only. They never need a bucket or a key.

## Open data

[data.nrgon.net](https://data.nrgon.net/) publishes reference data for Singapore,
Malaysia, Thailand, Vietnam, Indonesia and the Philippines: countries, utilities, tariff
structures, time-of-use schemes, national statistics and grid reliability. JSON over
HTTPS under `/v1/`, with a stated review cadence per dataset. Components read tariffs and
time-of-use bands from here rather than carrying a copy.

## Rules

- **Stored as received.** The raw message or file is kept unchanged. Parsing happens on
  read, so a parser fix applies to history.
- **Once.** A repeated message, identified by its own id, exists once.
- **Dated.** Every object carries the time it was received, in UTC, in its key.
- **Units stated.** A number without a unit is not stored.
- **Traceable.** A derived value names the objects it was computed from.