---
id: parallel-multi-point-jobs
status: not-started
depends_on: []
area: backend
description: Use bounded worker-local process parallelism for sweep, grid, and Roth explorer jobs that evaluate independent simulation points.
---

## Goal

Multi-point worker jobs currently evaluate independent simulation points serially inside one
claimed run: sensitivity/country sweeps loop over points, the What Would It Take grid loops
over 343 cells, and the Roth conversion explorer loops over baseline plus candidate schedules.
On a 2-CPU worker instance, those jobs should be able to use both CPUs without sharding a
normal single Monte Carlo run. Add bounded, opt-in process parallelism in the worker-owned
task layer while preserving existing response shapes, result ordering, tier gates, and the
runtime compute boundary.

## Acceptance Criteria

- [ ] Add a shared multi-point execution helper, likely under `simulation_engine/`, that resolves
      an env-controlled process count from `SIM_MULTI_POINT_JOB_PROCESSES` with default `1`,
      minimum `1`, and caps at both available CPU count and point count. The helper must use
      sequential execution whenever the resolved count is `1`.
- [ ] Parallel point workers must be top-level, pickle-safe functions that create their own
      `CoreSimulationRunner` inside the child process. Existing code paths that pass a custom
      `runner` or monkeypatch `simulate_and_collect_stats` must remain sequential so unit tests
      and fake runners continue to work.
- [ ] `run_sweep()` evaluates generated sweep points in parallel when eligible. The returned
      `points` array stays in the exact same logical order as today, `current_value` and country
      disclaimer behavior are unchanged, and any point failure fails the whole sweep run with the
      same worker failure path used today.
- [ ] `compute_exploration_grid()` evaluates grid cells in parallel when eligible. DB progress
      updates and cancellation checks stay in the parent worker process only; completed cells are
      counted as they finish, cancellation shuts down pending work best-effort, and the persisted
      `grid_points` ordering remains deterministic by the original retirement-year/spending/savings
      traversal order.
- [ ] `execute_roth_explorer()` evaluates the baseline and valid candidate simulation configs in
      parallel when eligible. Schedule generation, assumptions, warnings, max-candidate validation,
      candidate labels, and output shape remain unchanged and deterministic.
- [ ] Parallel execution must avoid fork-inherited duplicate random streams. The parent assigns a
      deterministic per-point seed before dispatch, and each child seeds both NumPy and Python
      `random` for its point. With `SIM_MULTI_POINT_JOB_PROCESSES=1`, seeded behavior remains
      byte-for-byte compatible with the current sequential path; with `>1`, a fixed parent seed
      produces reproducible parallel results even if they are not numerically identical to the
      sequential realization.
- [ ] Add focused tests for the helper and all three call sites: default sequential behavior,
      forced `SIM_MULTI_POINT_JOB_PROCESSES=2` order preservation, no parallel branch for custom
      runners/monkeypatched stats, and parent-only grid progress/cancellation behavior. Keep
      existing sweep/grid/Roth tests passing without changing their assertions unless they opt
      into the new env var.
- [ ] Add structured worker logs or low-cardinality metrics for multi-point jobs that record
      `run_kind`, point count, resolved process count, and wall-clock duration. Update
      `docs/render-deploy.md` and `render.yaml` to document how a 2-CPU worker should opt into
      `SIM_MULTI_POINT_JOB_PROCESSES=2` without also oversubscribing `SIM_WORKER_PROCESSES`.

## Out of Scope

- Sharding one normal `simulate_and_collect_stats_with_artifacts()` Monte Carlo run across
  processes.
- Rewriting the vectorized simulation core, tax engine, or result artifact materializer.
- Changing scenario counts, grid dimensions, sweep point generation, or Roth explorer candidate
  limits.
- Moving any simulation compute into FastAPI request handlers. The API continues to validate,
  enqueue, and read persisted results; workers own all sweep, grid, Roth explorer, and simulation
  compute.
- Frontend UI changes. Existing polling/progress displays should work against the same stored
  run status and result payloads.

## Danger Zones

- **RNG duplication after fork.** A naive process pool can give multiple points identical NumPy
  streams because child processes inherit the same global RNG state. Per-point seeding is required
  before every point evaluation.
- **Oversubscription.** `SIM_WORKER_PROCESSES=2` plus `SIM_MULTI_POINT_JOB_PROCESSES=2` can create
  four CPU-bound child workloads on a 2-CPU instance. Deployment docs should steer production
  toward either queue throughput or per-job latency unless the instance has enough CPUs for both.
- **Memory duplication.** Each process imports the simulation and tax stacks and materializes result
  payloads. Keep the default at `1`, cap process count conservatively, and do not parallelize
  tiny point sets where process overhead dominates.
- **Grid cancellation is cooperative.** A cancellation signal can only be observed by the parent
  between completed points. On cancellation or failure, shut down pending futures with
  `cancel_futures=True`; already-running child evaluations may finish and be discarded.
- **Pickle boundaries.** Do not pass database connections, progress callbacks, injected fake
  runners, or non-top-level closures to child processes. Pass only `SimulationConfig` snapshots,
  point descriptors, scalar options, and per-point seeds.

## Agent Notes

- Start with an internal helper instead of three ad hoc process pools. A useful shape is
  `run_points(kind, points, evaluate_point, max_processes=None)` returning indexed results in
  original point order while optionally reporting parent-side progress.
- Keep worker telemetry in the parent process. Child processes should not configure Sentry/OTLP or
  emit high-cardinality labels.
- For local benchmarking, compare `SIM_MULTI_POINT_JOB_PROCESSES=1` vs `2` for:
  - a normal sensitivity sweep,
  - a full What Would It Take grid,
  - a Roth explorer run with three valid candidate schedules.
  Record wall-clock time, worker RSS, and whether queue wait changed separately from execution
  duration.