Decoding TOON Back to JSON: Type Rules, Strict Mode and Validating LLM Output

Encoding JSON as TOON is the easy direction: the encoder knows every type. Decoding is where the rules matter, because TOON leaves most strings unquoted and the decoder has to decide what each bare token means. It matters even more when the TOON was written by a language model, which can miscount rows or stop early.

Anatomy of a tabular header

users[3]{id,name,role}:
  1,Alice,admin
  2,Bob,user
  3,Cara,user

Decoding that block produces an array of three objects. The two declarations in the header are redundant on purpose: they let the decoder check the body against them.

How bare values get their types

An unquoted token is interpreted as a number, boolean or null when it looks exactly like one, and as a string otherwise. Quotes always force a string. These results come from the reference decoder (@toon-format/toon 2.1):

The practical consequence: if you write TOON by hand, or ask a model to, any string that looks like a number or a keyword needs quotes. Encoders do this automatically. And as with JSON, identifiers larger than 2⁵³ should be carried as quoted strings.

What strict mode rejects

The decoder is strict by default. It refuses input where the body disagrees with the header or the layout is ambiguous. These are the messages you will actually see:

Passing { strict: false } relaxes these checks. With it, the two-row table that claims three rows decodes to two objects without complaint. That is occasionally what you want when salvaging data, and almost never what you want in a pipeline.

Why this matters for LLM output

Models produce structured output one token at a time, and the usual failure modes are quiet: the response hits the output limit and stops mid-list, a row drops a field, or the model decides to summarize the last few entries. With JSON, a truncated response at least fails to parse. A truncated CSV parses fine and gives you fewer rows.

TOON's declared lengths turn those quiet failures into loud ones. If the model wrote [50] and delivered 43 rows, strict decoding fails and you can retry instead of silently processing partial data.

import { decode } from '@toon-format/toon';

function parseModelOutput(text: string) {
  // models usually wrap the answer in a fenced block
  const match = text.match(/```(?:toon)?\n([\s\S]*?)```/);
  const body = (match ? match[1] : text).trim();
  try {
    return { ok: true as const, data: decode(body) }; // strict by default
  } catch (error) {
    return { ok: false as const, reason: (error as Error).message };
  }
}

const result = parseModelOutput(response);
if (!result.ok) {
  // feed the reason back: "Expected 50 tabular rows, but got 43"
  // gives the model something concrete to fix
}

Delimiters other than commas

When values often contain commas, the encoder can use a tab or a pipe as the delimiter. The choice is declared inside the brackets, so the decoder needs no configuration:

users[2|]{id|name}:
  1|Ada, Countess of Lovelace
  2|Bob

Here the comma in the first name is ordinary text and needs no quotes. If you paste TOON that was copied from a chat window, check that tab delimiters were not converted to spaces on the way.

What survives a round trip

• Preserved: all values and their types, object key order, array order, empty arrays and empty strings.
• Normalized: number spelling. 1.50 comes back as 1.5 and 1e3 as 1000, the same as a JSON parse and re-print.
• Not representable in either format: comments, dates as a distinct type, NaN and Infinity.

Decode and inspect

New to the format? Start with TOON format explained, which covers the syntax and when it is worth using.