Deterministic Safe JSON Repair & AST Patching
Malformed JSON is one of the most common causes of runtime crashes in API pipelines, webhooks, and microservices. The explosion of Large Language Models (LLMs) generating structured JSON outputs has drastically increased the occurrence of non-compliant syntax.
Standard JSON parsers (such as JSON.parse() in JavaScript or json.loads() in Python) are strictly binary: they either succeed completely or throw a fatal exception. FormJson Safe Repair bridges this gap by implementing resilient AST tokenization that detects and fixes structural defects without data loss.
01. Supported Defect Categories & Resolution Rules
| Defect Category | Source Malformed Input | Repaired RFC 8259 Output | Repair Heuristic |
|---|---|---|---|
| LLM Markdown Fences | ```json {"id": 1} ``` | {"id": 1} | Strips outer backtick blocks and conversational preamble. |
| Unquoted Keys | {name: "Alice", age: 30} | {"name": "Alice", "age": 30} | Encapsulates unquoted object property identifiers in double quotes. |
| Single-Quoted Strings | {'status': 'active'} | {"status": "active"} | Normalizes single quotation delimiters to double quotes while escaping interior double quotes. |
| Python Literals | {"ok": True, "data": None} | {"ok": true, "data": null} | Maps Python booleans and None to RFC 8259 lowercase keywords. |
| Trailing Commas | [10, 20, 30,] | [10, 20, 30] | Removes superfluous trailing commas preceding closing brackets. |
| Truncated Objects | {"users": [{"id": 1 | {"users": [{"id": 1}]} | Balances open brackets and braces according to depth call stack. |
02. AST Patching & Non-Destructive Diff Verification
Blind regex replacement on unstructured text often corrupts valid content (for instance, replacing single quotes inside apostrophe words like "don't" or mangling URL query strings). Safe Repair uses a resilient tokenizer that maintains state awareness:
Distinguishes whether a token appears inside a string literal, key position, array boundary, or comment block.
Maintains an LIFO stack of opened brackets, cleanly closing unterminated payloads from aborted LLM streams.
Provides a character-accurate visual diff before mutation, allowing engineers to verify every single repaired token.
03. Programmatic JSON Repair Scripts (Node.js & Python)
For backend microservices processing untrusted LLM outputs or raw log files, integrate these sanitization heuristics:
function cleanLlmJson(rawOutput) {
// 1. Strip markdown code fences
let cleaned = rawOutput.trim().replace(/^```(?:json)?/gm, '').replace(/```$/gm, '');
// 2. Strip single-line JS comments
cleaned = cleaned.replace(/(^|[^:])\/\/.*$/gm, '$1');
// 3. Remove trailing commas before closing brackets
cleaned = cleaned.replace(/,\s*([}\]])/g, '$1');
// 4. Quote unquoted object keys
cleaned = cleaned.replace(/([{,]\s*)([a-zA-Z0-9_]+?)\s*:/g, '$1"$2":');
return JSON.parse(cleaned);
}import ast, json
def repair_python_dict_to_json(malformed_str):
"""Safely parses Python dict strings and emits valid RFC 8259 JSON."""
try:
evaluated = ast.literal_eval(malformed_str)
return json.dumps(evaluated, indent=2)
except Exception:
sanitized = malformed_str.replace("True", "true").replace("False", "false").replace("None", "null")
return json.dumps(json.loads(sanitized), indent=2)
raw_python_dump = "{'status': 'success', 'code': 200, 'active': True}"
print(repair_python_dict_to_json(raw_python_dump))