Skip to contents

Synthetic mode lets one task run on synthetic inputs, with no upstream task and no real data. Every table of a SurveillanceSystem_v9 has a csdb::SyntheticTable, which holds the contract of the table and its fixed datasets. With synthetic mode on, cs9 fills the input tables of a task from them, and each switchable source returns its simulator.

This vignette builds a small surveillance system on SQLite and runs one task on synthetic data. Two other vignettes cover the levels below and above it:

  • csdb, vignette("synthetic-data", package = "csdb"): profiles, simulation and the SyntheticTable of one table.
  • norsyss.cs9, vignettes/synthetic-development.Rmd in the private repository niphr/norsyss.cs9: a late task of a real pipeline, run on synthetic inputs alone.

A small system

The system has two input tables and one output table. anon_vignette_cases holds weekly cases, partitioned by region. anon_vignette_population holds the population of each region. The databases are SQLite files under tempdir().

library(data.table)

db_dir <- file.path(tempdir(), "cs9-synthetic")
unlink(db_dir, recursive = TRUE)
dir.create(db_dir)
Sys.setenv(
  CS9_AUTO = "0",
  CS9_PATH = db_dir,
  CS9_DBCONFIG_ACCESS = "config/anon",
  CS9_DBCONFIG_DRIVER = "SQLite",
  CS9_DBCONFIG_DB_CONFIG = file.path(db_dir, "config.sqlite"),
  CS9_DBCONFIG_DB_ANON = file.path(db_dir, "anon.sqlite")
)
cs9::reload_db_config()
ss <- cs9::SurveillanceSystem_v9$new(name = "synthetic")

Attach a SyntheticTable to a table

add_table() and add_partitionedtable() take the argument synthetic. It is one of these:

  • NULL, the schema-only default.
  • a spec, list(generator = , dir = , invariants = , profile = ). Each element is optional.
  • a csdb::SyntheticTable, which cs9 stores as given.

cs9 builds the SyntheticTable of a spec from the contract of the table: its field types, its keys and its validator_field_contents. So the spec states no column twice. A generator is a function(profile, n, seed, world) that returns a data.table. This one gives every region every week of the world, and ignores n:

cases_generator <- function(profile, n, seed, world) {
  set.seed(seed)
  d <- CJ(isoyearweek = world$isoyearweek, region = world$region)
  d[, cases := rpois(.N, 20)]
  return(d[, .(isoyearweek, cases, region)])
}
ss$add_partitionedtable(
  name_access = "anon",
  name_grouping = "vignette",
  name_variant = "cases",
  name_partitions = c("north", "south"),
  column_name_partition = "region",
  field_types = c(isoyearweek = "TEXT", cases = "INTEGER", region = "TEXT"),
  keys = "isoyearweek",
  synthetic = list(
    generator = cases_generator,
    invariants = list(
      cases_nonnegative = function(d) {
        if (all(d$cases >= 0)) TRUE else "A count of cases is negative."
      }
    )
  )
)
ss$synthetic_tables$anon_vignette_cases$keys
#> [1] "isoyearweek" "region"

The keys of a partitioned table get the partition column, because two partitions can hold the same week. cs9 also adds the invariant partition_known, which refuses a region outside name_partitions.

The population table gets a generator with a defect. It writes the regions in title case, and ignores the world:

add_population <- function(generator) {
  ss$add_table(
    name_access = "anon",
    name_grouping = "vignette",
    name_variant = "population",
    field_types = c(region = "TEXT", pop = "INTEGER"),
    keys = "region",
    synthetic = list(generator = generator)
  )
}
add_population(function(profile, n, seed, world) {
  return(data.table(region = c("North", "South"), pop = 50000L))
})

set_synthetic()

Synthetic mode writes to the databases of the system. set_synthetic(TRUE) therefore refuses unless every database is a SQLite file under tempdir(). It checks each entry of config$dbconfigs and the database of every table. Here a production access is added to the configuration:

cfg <- cs9::config
cfg$dbconfigs$prod <- list(
  access = "prod",
  driver = "PostgreSQL Unicode",
  db = "surveillance"
)
ss$set_synthetic(TRUE)
#> Error:
#> ! Synthetic mode refuses config$dbconfigs: the driver is 'PostgreSQL Unicode', not SQLite. Synthetic mode MUST write only to a SQLite file under tempdir().

The check reads the configuration only. It opens no connection. Without the production access, the call passes:

cfg$dbconfigs$prod <- NULL
ss$set_synthetic(TRUE)
ss$synthetic_mode
#> [1] TRUE

set_synthetic_world()

The world is a named list of the allowed values of each column. Every generator gets it as world. One world serves every table. Generators that read it draw the same regions and weeks.

ss$set_synthetic_world(list(
  isoyearweek = c("2026-01", "2026-02", "2026-03", "2026-04"),
  region = c("north", "south")
))
print(ss$synthetic("anon_vignette_cases", n = NULL))
#> Key: <isoyearweek, region>
#>    isoyearweek cases region
#>         <char> <int> <char>
#> 1:     2026-01    17  north
#> 2:     2026-01    25  south
#> 3:     2026-02    25  north
#> 4:     2026-02    21  south
#> 5:     2026-03    13  north
#> 6:     2026-03    22  south
#> 7:     2026-04    23  north
#> 8:     2026-04    22  south

populate_synthetic() and the join check

The task reads the cases and the population, and writes the weekly rate per 100,000 to anon_vignette_rates.

ss$add_table(
  name_access = "anon",
  name_grouping = "vignette",
  name_variant = "rates",
  field_types = c(isoyearweek = "TEXT", region = "TEXT", rate = "DOUBLE"),
  keys = c("isoyearweek", "region")
)
print(ss$synthetic("anon_vignette_rates", n = 3))
#>    isoyearweek region      rate
#>         <char> <char>     <num>
#> 1:     2026-03  north 0.9446753
#> 2:     2026-03  south 0.6607978
#> 3:     2026-04  north 0.6291140

The output table has the schema-only default. Its keys take their values from the world, and rate is a uniform value from 0 to 1.

cs9 finds the functions of a task by name with get(), so they live in the global environment:

rates_data <- function(argset, tables) {
  # A partitioned table holds one table for each partition.
  cases <- lapply(tables$cases$tables, function(x) dplyr::collect(x$tbl()))
  return(list(
    cases = rbindlist(cases),
    population = dplyr::collect(tables$population$tbl())
  ))
}
rates_action <- function(data, argset, tables) {
  d <- merge(data$cases, data$population, by = "region")
  d[, rate := 100000 * cases / pop]
  tables$rates$drop_all_rows()
  tables$rates$insert_data(d[, .(isoyearweek, region, rate)])
}
assign("rates_data", rates_data, envir = globalenv())
assign("rates_action", rates_action, envir = globalenv())
add_rates_task <- function() {
  ss$add_task(
    name_grouping = "vignette",
    name_action = "rates",
    for_each_plan = plnr::expand_list(plan = 1),
    data_selector_fn_name = "rates_data",
    action_fn_name = "rates_action",
    tables = list(
      cases = ss$partitionedtables$anon_vignette_cases,
      population = ss$tables$anon_vignette_population,
      rates = ss$tables$anon_vignette_rates
    ),
    synthetic_joins = list(
      c("anon_vignette_cases", "anon_vignette_population")
    )
  )
}
add_rates_task()

populate_synthetic(task = ) fills every table in the tables of the task. cs9 cannot tell an input table from an output table, so name the inputs in tables.

The join check is off by default. add_task(synthetic_joins = ) declares join groups, and populate_synthetic(task = ) checks only these groups. Two tables of a group that share key columns MUST have a row in common on them. The task declares the cases and the population as one group. They share region, and the population holds no region of the world:

ss$populate_synthetic(
  task = "vignette_rates",
  tables = c("anon_vignette_cases", "anon_vignette_population"),
  n = NULL
)
#> Error:
#> ! populate_synthetic() refuses the task: the synthetic data of the tables anon_vignette_cases and anon_vignette_population have no row in common on their shared key columns region. Nothing was written.

The call checks every table and builds all the data before its first write, so a refusal writes nothing. The fix is a generator that takes the regions from the world. The task is added again, so that it holds the new table:

add_population(function(profile, n, seed, world) {
  return(data.table(region = world$region, pop = 50000L))
})
add_rates_task()
written <- ss$populate_synthetic(
  task = "vignette_rates",
  tables = c("anon_vignette_cases", "anon_vignette_population"),
  n = NULL
)
written
#>      anon_vignette_cases anon_vignette_population 
#>                        8                        2

written holds the number of rows written to each table. Each table gets its own seed, from seed and the md5 of the table name. Two tables then do not share a stream of random values. dataset = "<name>" fills each table whose catalogue holds that dataset with it, and every other table with random data.

The task now runs on synthetic inputs alone:

invisible(utils::capture.output(ss$run_task("vignette_rates")))
dplyr::collect(ss$tables$anon_vignette_rates$tbl())[, c("isoyearweek", "region", "rate")]
#> # A tibble: 8 × 3
#>   isoyearweek region  rate
#>   <chr>       <chr>  <dbl>
#> 1 2026-01     north     32
#> 2 2026-02     north     30
#> 3 2026-03     north     44
#> 4 2026-04     north     26
#> 5 2026-01     south     56
#> 6 2026-02     south     48
#> 7 2026-03     south     50
#> 8 2026-04     south     42

synthetic_grid_generator()

A generator often gives every series every week. synthetic_grid_generator(grid) builds that generator from the world. grid names world elements. The generator gives one row for each combination of their values, and csdb::simulate_from_profile() draws the other columns. It ignores n.

A world element can be a data.frame. Its rows stay together, so a location_code keeps its granularity_geo. synthetic_weeks() and synthetic_years() give the 12 time columns of csfmt_rts_data_v2 as such a data.frame. Each table gets the columns of an element that it holds:

ss$add_table(
  name_access = "anon",
  name_grouping = "vignette",
  name_variant = "weekly",
  field_types = c(
    region = "TEXT",
    isoyearweek = "TEXT",
    isoyear = "INTEGER",
    isoweek = "INTEGER",
    visits = "INTEGER"
  ),
  keys = c("region", "isoyearweek"),
  synthetic = list(
    generator = cs9::synthetic_grid_generator(c("region", "weeks"))
  )
)
ss$set_synthetic_world(list(
  isoyearweek = c("2026-01", "2026-02", "2026-03", "2026-04"),
  region = c("north", "south"),
  weeks = cs9::synthetic_weeks("2026-01", "2026-04")
))
print(ss$synthetic("anon_vignette_weekly", n = NULL))
#>    region isoyearweek isoyear isoweek visits
#>    <char>      <char>   <int>   <int>  <int>
#> 1:  north     2026-01    2026       1     34
#> 2:  north     2026-02    2026       2     69
#> 3:  north     2026-03    2026       3     73
#> 4:  north     2026-04    2026       4     41
#> 5:  south     2026-01    2026       1     37
#> 6:  south     2026-02    2026       2     19
#> 7:  south     2026-03    2026       3     27
#> 8:  south     2026-04    2026       4     19

The grid makes the rows unique, so the generator draws without the keys. finish = function(d, seed, world) can then set values that one table needs.

populate_synthetic(derive = )

A random draw of the rates does not follow the cases and the population. derive computes a table from the tables that the call drew. Element x = f replaces the data of table x with f(data, world, seed), where data holds the data of every table:

derive_rates <- function(data, world, seed) {
  d <- merge(data$anon_vignette_cases, data$anon_vignette_population, by = "region")
  return(d[, .(isoyearweek, region, rate = 100000 * cases / pop)])
}
# n = 1: the random draw of the rates needs one row only, because derive
# replaces it. The generators of the inputs ignore n.
ss$populate_synthetic(
  task = "vignette_rates",
  n = 1L,
  derive = list(anon_vignette_rates = derive_rates)
)
cases <- rbindlist(lapply(
  ss$partitionedtables$anon_vignette_cases$tables,
  function(x) dplyr::collect(x$tbl())
))
rates <- dplyr::collect(ss$tables$anon_vignette_rates$tbl())
both <- merge(cases, rates, by = c("isoyearweek", "region"))
nrow(both)
#> [1] 8
all.equal(both$rate, 100000 * both$cases / 50000)
#> [1] TRUE

cs9 runs the functions in list order, after every draw. It then validates every table again, so a function that breaks any table stops the call before the first write. A name of a table that the call does not fill stops the call before the first draw.

Put computed values in derive, not in a generator or a finish function. A csdb dataset stores the md5 of the data that the generator draws, and check_hashes() compares it on every platform. Functions such as pnorm(), qnorm() and exp() can differ in the last bit between Windows and Linux, so a generator that calls them can give a hash that passes on one platform and fails on the other. The output of derive is computed at populate time and has no stored hash.

add_source() and source()

In synthetic mode, a task MUST NOT read an external source, such as a register or an SQL Server. add_source() registers a reader and a simulator under one name. source(name) returns the simulator with synthetic mode on, and the reader otherwise. The two MUST have the same formal arguments.

ss$add_source(
  "register",
  reader = function(isoyearweek) stop("The register is not reachable here."),
  simulator = function(isoyearweek) data.table(isoyearweek = isoyearweek, cases = 3L)
)
ss$source("register")("2026-04")
#>    isoyearweek cases
#>         <char> <int>
#> 1:     2026-04     3

The call site reads ss$source("register")(isoyearweek) in both modes. A simulator with other arguments is refused:

ss$add_source(
  "register_2",
  reader = function(isoyearweek) NULL,
  simulator = function(week) NULL
)
#> Error:
#> ! reader and simulator of source register_2 MUST have the same formal arguments. reader has (isoyearweek), simulator has (week).

With synthetic mode off, the same call reaches the reader:

ss$set_synthetic(FALSE)
ss$source("register")("2026-04")
#> Error:
#> ! The register is not reachable here.

run-task.sh –synthetic

A detached run of a task of an implementation package can start in synthetic mode. run_task_detached(synthetic = TRUE) passes --synthetic to run-task.sh:

cs9::run_task_detached("my_task", package_dir = ".", synthetic = TRUE)

From a shell, the same run is:

RUN=$(Rscript -e 'cat(system.file("scripts", "run-task.sh", package = "cs9"))')
"$RUN" -d . --synthetic my_task

The child process calls set_synthetic(TRUE) and populate_synthetic(task = "my_task", dataset = "default") before run_task(). The log header then holds SYNTHETIC: yes. That call has no tables, so it fills every table of the task, outputs included. The join check covers only the join groups that the task declares.

A detached run has its own tempdir(). The implementation package MUST therefore point its CS9_DBCONFIG_DB_* variables under tempdir() in its .onLoad(), then call reload_db_config(). Otherwise set_synthetic(TRUE) refuses in the child and the run stops.