Data Backend

IronFlock provisions a private database for each project, powered by TimescaleDB. Your app defines the data schema; IronFlock creates the tables and starts collecting data the moment a device is added to the app.

How It Works

1. Define your data schema in .ironflock/data-template.yml.
2. Use the IronFlock SDK to publish data from your edge code.
3. IronFlock automatically sets up the database tables in each project where the app is installed.
4. Data flows from devices through the messaging system into the project database.

Each project gets its own physical database — there is no data sharing between projects.

The user has full control over the data collected by your app in their project. As a developer, you do not have access to this data.

Defining the Data Schema

Create a data-template.yml file in the .ironflock/ directory:

data:
  tables:
    - tablename: sensordata
      columns:
        - id: tsp
          name: Timestamp
          description: Timestamp of measurement
          path: args[0].timestamp
          dataType: timestamp
        - id: temperature
          name: Temperature
          description: Temperature reading in Celsius
          path: args[0].temperature
          dataType: numeric
        - id: humidity
          name: Humidity
          description: Relative humidity percentage
          path: args[0].humidity
          dataType: numeric
        - id: device_id
          name: Device ID
          description: Source device identifier
          path: args[0].device_id
          dataType: string

Column Options

Secret Columns

Some values have to be stored but must never be shown: an API token, a device password, a licence key. Mark the column secret: true and the data backend encrypts it on insert:

- tablename: credentials
  columns:
    - id: tsp
      dataType: timestamp
    - id: device_id
      dataType: string
    - id: api_token
      dataType: string
      secret: true

From then on no ordinary read returns the plaintext — not even to the app that wrote it:

NULL stays NULL everywhere, so “set but hidden” remains distinguishable from “never written”.

Reading a secret back in clear is only possible from the app’s own containers, through the SDK — see Secret Columns in the SDK reference. Boards and other apps cannot obtain it at all, which is what makes “never displayed in clear” true rather than merely cosmetic.

Updating a row without losing its secret. On an entity table, a client that edits a row can never send the real secret back — reads only ever gave it the placeholder. The data backend therefore treats the placeholder __secret__ (and the •••••••• mask boards display) as keep the previous value on write, and an omitted secret column keeps it too. Sending an explicit null clears the secret. Editing a machine’s description in a board form therefore never wipes its access code.

Two consequences follow: the literal strings __secret__ and •••••••• cannot themselves be stored as secret values (and raw ifsec:… ciphertext is rejected as input), and on a table without an entity key a placeholder write is rejected — there is no previous row to keep.

Three consequences are worth planning around, because they are hard limits rather than guidance:

• Secret columns are strings only. Encryption produces text, so numeric, boolean and timestamp columns cannot be secret. The mandatory tsp column cannot be secret either.
• You cannot filter, group or sort by a secret column. Every row is encrypted under its own random value, so two rows holding the same secret store different ciphertext. Equality filters, GROUP BY, ORDER BY and DISTINCT over a secret column simply cannot work. To answer “does this value match?”, use the SDK’s verify function, which compares inside the data backend instead of returning anything.
• A secret column cannot be an entity key. Because every row’s ciphertext differs, latest-per-entity reads would treat each row as its own entity, so naming one in maintainLatestFlagFor is rejected outright.

The last two restrictions are checked when your data template is validated, so a table that breaks them fails on release rather than misbehaving later.

Table Options

Besides columns, a table takes a few optional keys that control how it is described and how its data ages:

data:
  tables:
    - tablename: sensordata
      description: Environmental readings from the shop floor
      chunkTimeInterval: 1 hour
      dropAfter: 30 days
      columns:
        # ...

chunkTimeInterval controls how the time-series data is partitioned on disk. Pick it so that one partition roughly matches what you query at once: high-frequency data collected every second benefits from small chunks (minutes to hours), slow-moving data from large ones (weeks). This is only the app’s default — the project owner can adjust it later on their own data backend.

dropAfter turns the table into a rolling window. Whole partitions older than the given interval are dropped, which is far cheaper than deleting individual rows. The cleanup job runs on a dropAfter / 4 schedule, so a record can outlive its expiry by up to a quarter of the interval before its partition goes. Omit dropAfter to keep data indefinitely.

Both take PostgreSQL interval strings — 30 minutes, 1 hour, 7 days, 6 months.

Continuous Downsampling

Dashboards can ask the database to aggregate data — hourly averages, daily totals, counts per machine. Computing those from raw records is fine for a day and expensive for a year. Add downsample to a table and the platform maintains a continuously updated, pre-aggregated copy of it, then answers long-window queries from that copy instead:

data:
  tables:
    - tablename: sensordata
      dropAfter: 30 days
      downsample:
        bucket: 1 minute
        keepFor: 2 years
        paths:
          - payload.temperature
      columns:
        # ...

bucket is the finest resolution a chart can be served from — a chart asking for buckets much finer than this reads the raw table instead. It accepts fixed-width intervals from 1 second to 1 day that divide a day evenly (1 minute, 5 minutes, 1 hour). The default of 1 minute suits practically every dashboard; a coarser bucket costs less storage and less write throughput.

keepFor is what makes long histories possible at all. Raw records disappear with dropAfter, but the downsampled copy has its own retention: keep raw data for 30 days and downsampled data for 2 years, and a board can still chart two years of hourly averages on a fraction of the storage. Set it longer than dropAfter — the platform rejects the opposite as a misconfiguration.

paths extends downsampling to values inside JSON columns. Numeric columns are included automatically; JSON fields must be named explicitly, since a JSON column has no fixed set of keys. Undeclared fields still work in dashboards — they are simply computed from the raw table.

Everything else about it is automatic. Every numeric column gets its statistics maintained (average, sum, minimum, maximum, first, last and a record count), grouped by the table’s entity key (maintainLatestFlagFor, or the publishing device). Dashboards need no configuration and no awareness of it: a widget queries as usual, and the platform decides per query whether the pre-aggregated copy can answer it — falling back to the raw table transparently when it cannot, for example when a filter references a column the copy does not group by.

Schema changes rebuild the copy. Adding, removing or retyping a column of a downsampled table — or editing the downsample block itself — rebuilds the pre-aggregated copy from the raw table. Anything older than dropAfter cannot be reconstructed and is lost. Set the block up together with the table where you can, and treat later schema changes on long-lived tables as a deliberate decision.

Publishing Data from Edge Code

Use the IronFlock SDK to send data from your app:

from ironflock import IronFlock
 
flock = IronFlock()
 
flock.publish_to_table("sensordata", {
    "timestamp": "2025-01-15T10:30:00Z",
    "temperature": 23.5,
    "humidity": 62.1,
    "device_id": "sensor-001"
})

import { IronFlock } from "ironflock";
 
const flock = new IronFlock();
 
flock.publishToTable("sensordata", {
    timestamp: "2025-01-15T10:30:00Z",
    temperature: 23.5,
    humidity: 62.1,
    device_id: "sensor-001"
});

For high-frequency data, send many rows in a single message instead of one round-trip per row using publish_rows_to_table / publishRowsToTable (fire-and-forget) or append_rows_to_table / appendRowsToTable (returns the insert outcome). Each batch is inserted atomically — all-or-nothing. See the SDK reference for details.

Transform Tables

You can define SQL transforms that automatically aggregate or process your raw data:

data:
  tables:
    - tablename: sensordata
      columns:
        # ... raw data columns ...
 
  transforms:
    - tablename: hourly_averages
      materialize: true
      schedule: "0 * * * *"
      sql: >
        SELECT
          time_bucket('1 hour', tsp) AS hour,
          avg(temperature) AS avg_temp,
          avg(humidity) AS avg_humidity
        FROM sensordata
        GROUP BY hour
      columns:
        - id: hour
          name: Hour
          dataType: timestamp
        - id: avg_temp
          name: Average Temperature
          dataType: numeric
        - id: avg_humidity
          name: Average Humidity
          dataType: numeric

Transform tables are accessible in boards and via the SDK, just like regular tables.

Tracking the Latest State of an Entity

For tables that represent the current state of real-world entities — machines, assets, production orders — IronFlock supports a pattern called latest-state tracking.

Instead of overwriting a row when something changes, you always append a new row. You declare which columns identify a unique entity, and IronFlock derives the most recent row per entity whenever the table is read. This gives you a full history of every change while also making it easy to query only the current state.

Enable it on a table with maintainLatestFlagFor:

- tablename: machineform
  maintainLatestFlagFor: ['machinename']
  columns:
    - id: tsp
      dataType: timestamp
    - id: machinename
      dataType: string
    - id: machinetype
      dataType: string
    - id: active
      dataType: boolean
    - id: description
      dataType: string

maintainLatestFlagFor takes a list of columns that together identify a unique entity. Nothing is written into the row itself: IronFlock indexes the table by that entity key plus the timestamp, and picks the newest row per entity at query time. A row that arrives late or out of order can therefore never leave a stale marker behind.

Updating an Entity with a Partial Row

A row appended to an entity table is a new version of that entity — and it does not have to be complete. Columns the row does not provide are inherited from the entity’s previous latest row, so updating a single field means publishing just the entity key, a timestamp and that field:

Two details are worth knowing:

• A row that provides every column skips the previous-row lookup entirely, so complete rows stay exactly as cheap as they have always been — keep sending complete rows on high-frequency paths.
• Inheritance respects time: a row that arrives with an older timestamp (a backfill) inherits only from rows at or before its own tsp, never from newer ones.

Tables without maintainLatestFlagFor keep plain append semantics: a column the row does not provide is stored as NULL.

To query only current machine states:

SELECT DISTINCT ON (machinename) *
FROM machineform
ORDER BY machinename, tsp DESC

To view the full history of a specific machine:

SELECT * FROM machineform WHERE machinename = 'Assembly-Line-01' ORDER BY tsp

You rarely write that query by hand. Widgets on a board that connect to this table have a latest toggle in their filter settings, so users always see current values without any extra work. From the SDK, request the same mode by adding {"latest": true} to filterAnd — see getHistory.

Upgrading from latest_flag: earlier versions of IronFlock stored a physical boolean column called latest_flag. That column no longer exists — the current state is derived in SQL instead, which keeps it correct when rows arrive out of order. Existing boards and SDK calls that filter on latest_flag = true keep working: IronFlock recognises them and applies the latest-state mode. New code should use the latest toggle or the {"latest": true} filter entry.

Soft Deleting Records

IronFlock’s append-only model means records are never physically deleted. Instead, use a deleted boolean column to mark a record as removed. This preserves the full audit trail while hiding deleted records from dashboards.

Add a deleted column to any entity table:

- id: deleted
  name: Deleted
  dataType: boolean

When a user deletes a record (for example via a form on the board), your app publishes a new row for that entity with deleted: true. Combined with maintainLatestFlagFor, this new row becomes the latest state. Later partial rows inherit the deleted marker like any other column, so an update that does not mention deleted neither resurrects the entity nor hides it.

To query only active (non-deleted) current records:

SELECT * FROM (
  SELECT DISTINCT ON (machinename) *
  FROM machineform
  ORDER BY machinename, tsp DESC
) latest
WHERE deleted IS NULL OR deleted = false

The deleted check runs after the latest row per machine has been picked. That order matters: filtering deleted rows out first would make the previous, non-deleted row resurface as the current state.

Board widgets and the SDK apply the same order automatically — combine the latest toggle (or {"latest": true}) with a deleted filter and you get exactly this behaviour. Deleted records disappear from the dashboard immediately after the form is submitted, but remain in the database for history and audit purposes.

Sharing Data with Other Apps

Your data backend is private to your app: no other app installed in the project can see your tables. Two optional keys in data-template.yml change that.

To read another app’s data, list the apps you want to read from in a top-level consumes: section — next to data:, not inside it:

consumes:
  - app: machine-monitor
    reason: "Computes OEE from the monitor's machine state and counter streams"
 
data:
  tables:
    - tablename: oee_results
      columns:
        # ... your app's own tables, as usual

app is the technical name of the providing app, or "*" (quotes required) for every app in the project. reason is shown to the user in the consent dialog — the declaration alone grants nothing until they approve it.

To keep individual tables back, mark them private: true. Everything you define is shareable by default; a private table or transform never appears in the catalog other apps see.

data:
  tables:
    - tablename: measurements        # shared (default)
      columns: [ ... ]
    - tablename: calibration_state   # internal — never visible to other apps
      private: true
      columns: [ ... ]

Access is read-only, granted per project by the user, and revocable at any time. See Consuming Data from Other Apps for the full model and the SDK calls that read a provider’s history and live streams.