Skip to content

Out-of-Band Delimiters for Shell IPC Pipelines

Pattern

When a shell script's internal IPC uses a single-character record delimiter (typically | or : for cut/IFS/awk -F), the delimiter MUST be a character the data domain cannot contain. If the data domain is "arbitrary user input that may later appear in Markdown tables," that rules out |.

Use ASCII Unit Separator (\037, octal 037, hex 0x1F) as the IPC delimiter. It is: - Non-printable, so no human ever types it - Already designed for this purpose (ASCII control codes 0x1C–0x1F are File/Group/Record/Unit Separators) - Safe for IFS="$DELIM" read -r and awk -F"$DELIM"

Defence in Depth

Two layers, both required:

  1. IPC layer: Use \037 as the record delimiter end-to-end. Every emitter and every reader must agree.
  2. Render layer: Escape the user-facing delimiter (| for Markdown) at the boundary where the value enters the rendered output. Even if a stray | sneaks past layer 1 (e.g. via a future code path that forgets), the render still produces valid output.

Layer 2 alone is fragile — easy to miss a render path. Layer 1 alone is fragile — easy for a future contributor to fall back to cut -d'|'. Together they are robust.

Regression Test

Pin the invariant with a test whose fixture deliberately contains the dangerous character:

make_state "C001" 5 "Foo" "P1" "specified" "Alice|Bob" "plan"
output=$(bash aggregate.sh VisiMatch)
assert_contains "$output" "| Alice\|Bob | plan |"

Without the fixture explicitly containing |, the test cannot catch a regression to cut -d'|'.

In SPEC-120

  • Bug: assignee names like Alice|Bob would render as a 7-cell row instead of 6 in the dashboard Markdown.
  • Fix commit: 3682576b.
  • Layer 1: DELIM=$'\037' constant (aggregate.sh:89) applied to every emitter (parse_projects_yaml, parse_design_state, INPROGRESS/SPECIFIED/COMPLETED writes) and every reader (while IFS="$DELIM" read).
  • Layer 2: escape_pipe() helper (aggregate.sh:102–104) called on every cell in render_inprogress_row, render_specified_row, render_completed_row.
  • Regression test: G3-5 in test-spec-120-domain-dashboard.sh:335–354.

When to Apply

Any shell script that: - Uses single-character record delimiters for IPC between functions or subshells - Aggregates values from external sources (YAML, git author names, config files, JSON) - Renders any of those values into a downstream syntax with reserved characters (Markdown tables |, CSV ,, TSV \t, JSON ")

The delimiter and the rendered syntax's reserved character should be different by construction. If they overlap, escape at render and move IPC out of band.

Why Not Just Escape | in IPC?

Escaping | to \| for IPC means every cut/IFS reader needs awareness of the escape rules, and round-trips become error-prone. Picking a delimiter the data domain cannot contain is simpler and safer than designing an escape protocol for a delimiter the data domain frequently does contain.