--- description: View the status of all running experiments; optionally auto-collect completed experiments and advance the pipeline argument-hint: "[--pipeline ] [--collect-ready] [--auto-advance]" --- # /exp-status > Unified experiment status monitoring entry point. > Scans all `running` experiments, performs a live status check on each (screen session / SSH), > and outputs a status table (alive / anomaly / completed) to guide the user's next actions. > > When used with `/research --auto`, acts as a periodic checker scheduled by CronCreate: > when all experiments in a pipeline are completed, automatically triggers `/research --start-from stage4`. ## Inputs - No arguments (default): check all `running` experiments, print status table - `--pipeline ` (optional): check only experiments belonging to the specified pipeline; additionally print overall pipeline progress - `--collect-ready` (optional): auto-call `/exp-run --collect` for all experiments whose session has already ended - `--auto-advance` (optional, requires `--pipeline `): if all pipeline experiments are `completed`, automatically trigger `/research --start-from stage4` without waiting for the user ## Outputs - **Status report** (terminal output, all modes): list of experiments in running/anomaly/completed states - `wiki/experiments/{slug}.md` — updated (outcome/key_result/status) when `--collect-ready` triggers Phase 4 - `wiki/outputs/pipeline-progress.md` — `--auto-advance` updates current_stage → stage4 (done internally by /research --start-from stage4) - `wiki/log.md` — appended status check log ## Wiki Interaction ### Reads - `wiki/experiments/*.md` — status, remote frontmatter (server/session/started), date_planned - `wiki/outputs/pipeline-progress.md` — in `--pipeline` mode, identifies target experiments and monitoring_cron_id ### Writes - `wiki/experiments/{slug}.md` — updated via /exp-run --collect in `--collect-ready` mode - `wiki/outputs/pipeline-progress.md` — updated by /research when `--auto-advance` triggers Stage 4 - `wiki/log.md` — appended status check log ### Graph edges created - None (result writes triggered indirectly via /exp-run --collect do not produce new edges) ## Workflow **Precondition**: confirm working directory is the wiki project root (directory containing `wiki/`, `raw/`, `tools/`). ### Step 1: Collect Target Experiment List 1. **Determine check scope**: - If `--pipeline ` is specified: - Read `wiki/outputs/pipeline-progress.md`, extract the slug list from the `stage3a_deployed` field - If the file does not exist or slug does not match: report error, suggest running `/research` first or specifying manually - Otherwise: - Use Glob to scan `wiki/experiments/*.md`, filter for `status == running` 2. **If no running experiments**: - Print a friendly message: ``` No running experiments found. - To start an experiment: /exp-run - To see all experiments: check wiki/experiments/ ``` - Return ### Step 2: Check Status of Each Experiment For each target experiment, execute in parallel (or sequentially): 1. **Read experiment page**: from `wiki/experiments/{slug}.md` get: - `remote` block (if present, this is a remote experiment) - `run_log` path - `started` (from `remote.started` or `date_planned`, used to compute elapsed time) - Deployment environment (has remote block → remote, otherwise → local) 2. **Check process status**: - **Local**: `screen -ls | grep "exp-{slug}"` - Has output → `alive: true` - No output → `alive: false` (session is gone) - **Remote**: `python3 tools/remote.py check --name "exp-{slug}"` - Parse JSON: `alive`, `last_lines`, `anomalies` 3. **If alive == true**: - Fetch recent logs (at most 20 lines): - Local: `tail -20 {run_log}` - Remote: use `last_lines` from the `check` command response - Extract latest metric (loss, accuracy, step, etc. — grep the last metric line) - Detect anomalies (NaN/OOM/Traceback/Inf): use `anomalies` field from `remote.py check` (remote), or manual grep (local) - Compute elapsed time (current time − started) - Classify as: `running` or `anomaly` 4. **If alive == false**: - Classify as: `completed_pending_collect` (session gone but wiki status is still running) - If wiki status is already `completed`: classify as `collected` 5. **Aggregate results**: build status dict `{slug: {state, elapsed, latest_metric, anomalies}}` ### Step 3: Print Status Report ```markdown # Experiment Status — {YYYY-MM-DD HH:MM} ### 🔄 Running ({N}) | Experiment | Elapsed | Latest | Env | |-----------|---------|--------|-----| | [[exp-foo-baseline]] | 2.3h | loss: 0.42 | local | | [[exp-foo-validation]] | 1.1h | step: 1200 | remote (gpu1) | ### ⚠️ Anomaly Detected ({N}) | Experiment | Elapsed | Issue | Action | |-----------|---------|-------|--------| | [[exp-foo-ablation]] | 0.8h | NaN loss at step 500 | Run `/exp-run exp-foo-ablation --collect` to inspect | ### ✅ Completed — Pending Collect ({N}) | Experiment | Finished (estimate) | |-----------|---------------------| | [[exp-foo-sanity]] | session gone | ### 📦 Already Collected ({N}) | Experiment | Outcome | |-----------|---------| | [[exp-foo-old]] | succeeded | --- ### Actions ```bash # Collect all completed experiments at once: /exp-status --collect-ready # Collect a specific experiment: /exp-run exp-foo-sanity --collect # Pipeline progress (if in /research): /exp-status --pipeline {pipeline-slug} ``` ``` Append log: ```bash python3 tools/research_wiki.py log wiki/ \ "exp-status | running: {N}, anomaly: {M}, pending-collect: {K}" ``` ### Step 4: --collect-ready Auto-Collect (if specified) For each `completed_pending_collect` experiment, call `/exp-run --collect`: ``` Skill: exp-run Args: "{slug} --collect" ``` Collect each completed experiment sequentially (not in parallel, to avoid concurrent wiki writes). After all collections are done, re-print the updated status report. ### Step 5: --auto-advance Pipeline Advance (if both --pipeline and --auto-advance are specified) 1. **Check pipeline completion condition**: - Read `stage3a_deployed` list from `wiki/outputs/pipeline-progress.md` - Check the status of each slug's `wiki/experiments/{slug}.md` - **Condition met**: all experiments have status == `completed` 2. **If condition is not met** (some experiments still running or pending collect): - Print current progress: `Pipeline {slug}: {M}/{N} experiments completed` - Return (do not advance) - Cron will trigger again in 30 minutes 3. **If condition is met (all experiments completed)**: a. **Print notification and trigger Stage 4**: - Print: ``` ✅ All experiments completed for pipeline {slug}! Advancing to Stage 4 (Verdict & Iteration)... ``` - Append log: ```bash python3 tools/research_wiki.py log wiki/ \ "exp-status | pipeline {slug}: all experiments done, advancing to stage4" ``` - Trigger next stage: ``` Skill: research Args: "--start-from stage4" ``` ## Constraints - **Read-only in non --collect-ready mode**: without `--collect-ready`, do not modify any wiki files - **`--auto-advance` requires `--pipeline`**: using `--auto-advance` alone is invalid, report an error - **Status checks must be non-blocking**: each experiment check should complete quickly (single SSH check or screen -ls) - **Anomalies are not auto-fixed**: `/exp-status` only reports anomalies; fixes require the user to manually call `/exp-run --collect` - **pipeline-progress.md must exist**: in `--pipeline` mode, if the file is missing, report an error ## Error Handling - **No running experiments**: print friendly message, not an error; provide next step suggestions - **`--pipeline` but pipeline-progress.md does not exist**: report error "Pipeline progress file not found. Run `/research ` first or check wiki/outputs/" - **`--auto-advance` without `--pipeline`**: report error "--auto-advance requires --pipeline " - **SSH connection fails** (remote experiment): mark that experiment as `check_failed`, note it in the report, continue checking other experiments - **screen -ls returns nothing**: does not mean the experiment failed — may be a brief delay; mark as `completed_pending_collect` - **`/exp-run --collect` fails** (`--collect-ready` mode): record the failure, continue collecting other experiments, report all failures at the end ## Dependencies ### Skills（via Skill tool） - `/exp-run` — call collect phase in `--collect-ready` mode - `/research` — trigger Stage 4 via `--auto-advance` ### Tools（via Bash） - `python3 tools/remote.py check --name "exp-{slug}"` — remote experiment status check - `python3 tools/remote.py tail-log --name "exp-{slug}" --lines 20` — fetch remote logs - `python3 tools/research_wiki.py set-meta ` — update pipeline-progress - `python3 tools/research_wiki.py log wiki/ ""` — append log - `screen -ls` — local process status - `tail -20 {log}` — fetch local logs ### Claude Code Native - `Read` — read experiment pages and pipeline-progress - `Write` — update pipeline-progress status - `Glob` — scan wiki/experiments/*.md - `Bash` — screen/tail and other system commands - `Skill` — call /exp-run --collect and /research ### Called by - CronCreate schedule (created by `/research --auto` Stage 3b: triggers every 30 minutes) - User directly - `/research` Stage 3b (in interactive mode, suggested to user)

/exp-status

Install this skill

Comments