DB Markham's Homepage

our-story-so-far

9 min read

Our Story So Far

What This Is

An Obsidian-based workflow automation system. It generates structured, multi-step workflow templates with rotation-based lookups, phrase injection, parallel tracks (splits/joins), and output-to-input carry-forward. Everything runs inside Obsidian using Templater and a custom workflow-tools plugin.

The project grew out of “The Review,” a newspaper-style editorial workflow with 20 mega-categories and 7 story angles rotating daily via day-of-year modulo. The system is now generic — any workflow with sequential and parallel steps can be generated from a config object.

Current State (Session 12, 2026-02-17)

  • 506 tests across 26 suites, all passing
  • Runtime modules in scripts/workflow/ (8 files)
  • Build tool + tests in tests/ (27 files)
  • workflow-creator.js lives in BOTH scripts/workflow/ AND tests/ (see Known Issues)
  • Trial workflow tr1-select-articles partially working:
    • Step templates generate and auto-strip correctly
    • Carry-forward works (output→input)
    • Home template crashes with yamlStr is not defined (old template needs deletion + regeneration)
    • Rotation files were accidentally deleted; recreated from original doySentence.js data
    • Phrase files not yet created for this workflow

Mistakes We Made and How to Avoid Them

This section exists so we stop repeating ourselves. Every entry is a real mistake that cost real time.

1. Templater executes <% tags everywhere, including inside code fences

What happened: The Quick Reference callout in the home template had code examples with <% yamlStr(...) %>. Templater executed them as real code even though they were inside markdown code blocks inside a callout.

Cost: Multiple failed workflow runs across multiple sessions. The home template crashed every time.

Rule: Never put <% or %> in ANY generated template content, even in code blocks, comments, or documentation callouts. Use backtick-escaped descriptions instead, or point to a separate static .md file.

2. Testing in Node ≠ running in Obsidian/Templater

What happened: Tests pass in Node.js because Node handles require(), module.exports, and standard JavaScript normally. Obsidian’s Templater has different module loading, different error handling, and executes template tags that Node never sees.

Cost: Repeatedly shipping code that passes all tests but crashes in Obsidian.

Rule: Node tests catch logic errors. They cannot catch Templater parsing errors. After ANY change to generated template content, the very next step is to test in Obsidian — not just run Node tests. We need a checklist item for this.

3. The require() path problem

What happened: workflow-creator.js needs to live in scripts/workflow/ (for Templater) AND in tests/ (because test files do require('./workflow-creator')). When we “fixed” the placement by removing it from one location, we broke the other.

Cost: 11 test files failed with “Cannot find module” errors. On the other side, create-workflow.md couldn’t find the module.

Rule: workflow-creator.js lives in both places until we fix the test require paths (backlog item). When updating, copy to both. The proper fix is changing tests to require('../scripts/workflow/workflow-creator').

4. Generated templates persist after the generator is fixed

What happened: Fixed the yamlStr bug in workflow-creator.js. Regenerated the workflow. But hit Enter (not “yes”) on the home template overwrite prompt, so the OLD buggy home template stayed in the vault.

Cost: Multiple more failed runs while debugging a “fixed” bug.

Rule: When regenerating a workflow, always say “yes” to EVERY overwrite prompt. If you accidentally skip one, delete that specific template file and re-run. Better yet: delete all master templates first (see Clean Reset below), then generate fresh.

5. Deleting too much during a reset

What happened: While cleaning up to reset tr1-select-articles, deleted the entire directory including the rotation lookup files (mega-categories, angles) which were hand-authored data, not generated templates.

Cost: Had to recreate rotation files from the original doySentence.js source.

Rule: Know what’s generated vs. what’s authored. The Clean Reset section below spells this out. Rotation files and phrase files are YOUR DATA. Template files are generated and disposable.

6. Saying a file is “safe” without tracing dependencies

What happened: Confidently stated workflow-creator.js could be removed from tests/ because “it has no require() statements.” Didn’t check that 11 test files require IT.

Cost: 11 broken tests, user frustration, loss of trust.

Rule: Before moving or removing ANY file, run grep -r "require.*filename" tests/ scripts/ to find all dependents. Add this to the pre-flight checklist.

7. The overwrite prompt defaults to “no”

What happened: Obsidian’s file creation prompt defaults to not overwriting. Hitting Enter without reading means “keep the old file.” With 40+ files in a workflow, it’s easy to accidentally keep stale templates.

Cost: Stale templates mixed with new ones. Inconsistent behavior. Hard to debug.

Rule: Until we build “Yes to All” (backlog), go slowly through overwrite prompts. Or better: delete all master templates first, then generate clean.

Pre-Flight Checklist

Run this before declaring any change “done”:

[ ] All Node tests pass (node tests/*.test.js)
[ ] workflow-creator.js copied to BOTH scripts/workflow/ AND tests/
[ ] No <% or %> tags in any generated template content (except step setup blocks)
[ ] grep -r "require.*changed-file" to find all dependents
[ ] If template content changed: test in Obsidian (not just Node)
[ ] If generated files changed: delete old templates, regenerate, say YES to all overwrites
[ ] Restart Obsidian after changing anything in scripts/workflow/

How to Clean-Reset a Workflow

When things get tangled and you need to start fresh.

Delete (generated files — disposable)

  1. The entire instance subfolder: {workflow-name}/{yyyy-mm}/
  2. All master templates: {workflow-name}-{dateStamp}-*.md
    • *-workflow.md (orchestrator)
    • *-home.md (dashboard)
    • *-01.md, *-02a.md, etc. (step templates)
    • *-phrases.md (instance phrase file, if dated)

KEEP (your authored data — not disposable)

  • {workflow-name}-rotation-*.md (rotation lists YOU wrote)
  • {workflow-name}-phrases.md (master phrase file YOU wrote, no dateStamp in name)
  • Everything in scripts/workflow/ (runtime modules)
  • Everything in tests/ (test suite)

Then

  1. Run create-workflow.md
  2. Say YES to every overwrite prompt
  3. Restart Obsidian
  4. Re-run the workflow template to generate instance files

Timeline

Session 1 — Obsidian Workflow Dated Paths (2026-02-12)

  • TDD approach established
  • Dated folder structure: workflow-name/yyyy-mm/
  • Path-building utilities, Templater collision handling

Session 2 — Dated Paths Integration (2026-02-12)

  • Integrated dated folders into workflow-creator.js
  • Filename collision prevention, recursive folder creation
  • 79 tests across 5 suites

Session 3 — Plugin TDD Loop (2026-02-13)

  • workflow-tools plugin: silent load failures, ES6 compat
  • Auto-strip functionality for template artifacts

Session 4 — Step Tracking Bidirectional Tests (2026-02-15)

  • Status/started/completed frontmatter fields
  • Dataview dashboard, plugin commands for status updates
  • Bidirectional testing, scripts folder reorganization

Session 5 — Splits/Joins Planning (2026-02-16)

  • Design session for parallel tracks and carry-forward
  • Confirmed join behavior (concatenate outputs labeled by track)

Session 6 — Splits/Joins Implementation (2026-02-16)

  • computeStepShape for track fan-out/fan-in
  • Integration with generateWorkflow

Session 7 — Carry-Forward Implementation (2026-02-16)

  • Output → input carry-forward module
  • Plugin wiring for file-watch triggers

Session 8 — Phrase Lookup & Date Utils (2026-02-16)

  • parsePhraseFile with scoped pools
  • date-numbers.js, create-workflow.md tracks support

Session 9 — Rotation Resolver & Phrase Wiring (2026-02-17)

  • rotation-resolver.js: modulo-based item selection from markdown
  • 10-track workflow support, migration from doySentence.js

Session 10 — Templater Fix & Date Folders (2026-02-17)

  • Fixed require() incompatibility in Templater
  • yyyy-mm-dd master template folders with dynamic instance subfolders

Session 11 — Phrase Sub-Categories & Rotation Mode (2026-02-17)

  • Sub-headed categories (### Goal, Motivation)
  • Per-workflow phraseMode config (random vs rotate)
  • Obsidian boot loop debugging (workspaces-plus plugin)

Session 12 — How-To Doc, Consolidation, Mistakes (2026-02-17)

  • workflow-lookup-howto.md reference document
  • our-story-so-far.md with mistakes documentation
  • Fixed yamlStr in home template (Templater executing code examples)
  • Confirmed workflow-creator.js is safe in scripts/workflow/
  • Identified dual-location requirement (scripts/ + tests/)
  • Recreated rotation files from original doySentence.js
  • 506 tests across 26 suites

Architecture

Runtime Modules (scripts/workflow/ — loaded by Templater)

ModulePurpose
rotation-resolver.jsParse rotation markdown, select items by day-of-year modulo
phrase-lookup.jsParse phrase markdown, scoped pools, random or modulo selection
date-numbers.jsDay-of-year calculation
dependency-checker.jsValidate required modules at startup
author-config.jsAuthor name/email for YAML
carry-forward.jsOutput → input content propagation
core-utils.jsShared utilities
template-factory.jsTemplate generation helpers
workflow-creator.jsGenerates all template files from config

Test Suite (tests/ — Node.js only)

FilePurpose
workflow-creator.jsCOPY of the same file (tests require it locally)
26 *.test.js filesFull regression suite, 506 tests

Key Concepts

Two selection modes: Modulo rotation (deterministic cycling) and random. Rotation is the default for lookups, random is the default for phrases.

Phrase cascade: Workflow → Instance → Base step → Exact step (track). All matching scopes merge into one pool.

Phrase sub-categories: ### Goal under ## Step 01 creates context.phrases["Goal"].

Splits/joins: Steps can fan out to parallel tracks (02a-02h) and fan back in.

Carry-forward: When a step completes, its output section flows into the next step’s input.

Backlog

High Priority

  • Reset workflow command — Plugin command to delete all generated files for a workflow while preserving rotation/phrase data. Shows file list, asks for confirmation.

  • “Yes to all” on overwrite prompts — Plus a “Back” option for accidental Enter presses.

  • Fix test require paths — Change all 11 test files from require('./workflow-creator') to require('../scripts/workflow/workflow-creator'). Eliminates the dual-location problem.

Medium Priority

  • Folder pre-creation — Workflow template should create the instance subfolder before writing the home file. Currently step 01 creates the folder as a side-effect, but home runs first and fails.

  • Rotation file naming guard — Warn when rotation files don’t match the workflow name pattern.

  • Obsidian smoke test script — A template that exercises all the template features (rotations, phrases, yamlStr, etc.) and reports pass/fail in the console. Catches Templater-specific bugs that Node can’t.

Low Priority / Future

  • Configurable rotation seed — Allow any integer as the modulo input, not just day-of-year.
  • Workflow diff/compare — Show what changed between generated versions.
  • Git integration — Auto-commit after major workflow operations.
  • Step template hot-swap — Replace a step’s setup block in-place.

Ideas (Unscoped)

  • Web-based workflow visualizer (static HTML, loads JSON)
  • Workflow-level phrase mode override per-step
  • Instance-level rotation overrides

Known Issues

  1. workflow-creator.js dual location — Must exist in BOTH scripts/workflow/ (for Templater) AND tests/ (for test suite). When updating, copy to both. Fix: change test require paths (backlog item).

  2. workspaces-plus plugin — Can corrupt workspace state and cause Obsidian boot loops. If Obsidian won’t start and console shows applyCss is not a function, delete .obsidian/plugins/workspaces-plus and workspace.json.

  3. Old template artifacts — Templates generated before recent fixes may have stale code (missing yamlStr, executable code examples in callouts). Fix: delete the specific template file and regenerate.

  4. rotation-lookup.js vs rotation-resolver.js — rotation-lookup.js is the old module. rotation-resolver.js is the new one. Both may exist in scripts/workflow/. The old one should eventually be removed.

  5. Overwrite prompt defaults to “no” — Hitting Enter without reading keeps the old file. Always explicitly type “yes” until we build “Yes to All.”

  6. Node tests can’t catch Templater bugs — Template parsing errors (executing <% in unexpected places) only manifest in Obsidian, not in Node test runs.

File Inventory for tr1-select-articles

Authored data (KEEP)

  • tr1-select-articles-rotation-mega-categories.md — 20 mega-categories from original doySentence.js
  • tr1-select-articles-rotation-angles.md — 7 story angles from original doySentence.js
  • tr1-select-articles-phrases.md — Master phrases (create when ready)

Generated templates (DISPOSABLE — regenerate with create-workflow.md)

  • tr1-select-articles-2026-02-workflow.md
  • tr1-select-articles-2026-02-home.md
  • tr1-select-articles-2026-02-01.md
  • tr1-select-articles-2026-02-02a.md through 02h
  • tr1-select-articles-2026-02-03a.md through 03h
  • tr1-select-articles-2026-02-04a.md through 04h
  • tr1-select-articles-2026-02-05a.md through 05h
  • tr1-select-articles-2026-02-06a.md through 06h
  • tr1-select-articles-2026-02-07.md
  • Plus: data, lookup, step-sections, step-content, step-custom .js files

Generated instances (DISPOSABLE)

  • Everything in tr1-select-articles/2026-02/

Transcripts

All session transcripts: /mnt/transcripts/ Journal index: /mnt/transcripts/journal.txt

Graph View