One of the many new features announced at yesterday’s OpenAI dev day is better support for generating valid JSON output.

From the JSON mode docs:

A common way to use Chat Completions is to instruct the model to always return JSON in some format that makes sense for your use case, by providing a system message. This works well, but occasionally the models may generate output that does not parse to valid JSON.

To prevent these errors and improve model performance, when calling gpt-4-1106-preview or gpt-3.5-turbo-1106, you can set response_format to { type: "json_object" } to enable JSON mode. When JSON mode is enabled, the model is constrained to only generate strings that parse into valid JSON.

In the past you could ask ChatGPT to generate JSON or use function calling to output JSON, but in my experience working with both approaches on Preceden, both methods would still occasionally return invalid JSON (due to characters not being escaped properly, for example). So, this new JSON mode is a welcome addition and will simplify quite a bit of my prompting.

Here’s a simple Ruby script demonstrating how to use JSON mode with GPT-4 Turbo:

	require "openai"
	unless ENV["OPENAI_SECRET_KEY"]
	puts "Please set the OPENAI_SECRET_KEY environment variable on the command line:"
	puts "$ export OPENAI_SECRET_KEY=\"sk-…\""
	return
	end
	def get_json(prompt)
	client = OpenAI::Client.new(access_token: ENV["OPENAI_SECRET_KEY"])
	response = client.chat(parameters: {
	model: "gpt-4-1106-preview",
	messages: [
	{ role: "user", content: prompt }
	],
	response_format: { type: "json_object" }
	})
	if response.key?("error")
	error_message = response.dig("error", "message")
	puts "Error: #{error_message}"
	return
	end
	response.dig("choices", 0, "message", "content")
	end
	prompt = <<STR
	Generate JSON about Bill Gates:
	{
	"full_name": "",
	"title": ""
	}
	STR
	puts get_json(prompt)

view raw json_mode.rb hosted with ❤ by GitHub

This instructs GPT-4 Turbo to generate valid JSON by setting the response_format to { type: "json_object" }. And sure enough, ChatGPT fills in the JSON values:

	{
	"full_name": "William Henry Gates III",
	"title": "Co-founder of Microsoft Corporation"
	}

view raw output.json hosted with ❤ by GitHub

Let’s try some other prompt variations and see what happens.

Not mentioning JSON in the prompt

	Generate details about Bill Gates:
	{
	"full_name": "",
	"title": ""
	}

view raw prompt.txt hosted with ❤ by GitHub

This will return an error:

‘messages’ must contain the word ‘json’ in some form, to use ‘response_format’ of type ‘json_object’.

Because as the docs note:

To use JSON mode, your system message must instruct the model to produce JSON. To help ensure you don’t forget, the API will throw an error if the string "JSON" does not appear in your system message.

Lowercase JSON

Does JSON have to be properly cased in the prompt?

	Generate json about Bill Gates:
	{
	"full_name": "",
	"title": ""
	}

view raw prompt.txt hosted with ❤ by GitHub

Lowercase is fine:

	{
	"full_name": "William Henry Gates III",
	"title": "Co-founder of Microsoft Corporation"
	}

view raw lowercase.json hosted with ❤ by GitHub

Describing the JSON

Instead of giving it a JSON template, what if we just described what we want?

Generate json about Bill Gates with two keys: full_name and title.

That works too:

Generating an array

Generate json about Bill Gates with three keys: full_name, title, and facts. facts should be an array with 3 items.

	{
	"full_name": "William Henry Gates III",
	"title": "Co-founder of Microsoft Corporation",
	"facts": [
	"Bill Gates started Microsoft with his childhood friend Paul Allen in 1975.",
	"He held the position of chairman, CEO, and chief software architect at Microsoft during his career.",
	"Beyond technology, Gates is also known for his philanthropic work with the Bill & Melinda Gates Foundation, which focuses on health, education, and poverty alleviation."
	]
	}

view raw facts.json hosted with ❤ by GitHub

Nice.

Nested JSON

Lets try generating nested JSON:

Generate json about Bill Gates with two three keys: full_name, title, and events.

Facts should be an array of 3 objects representing important milestones in his life. Each of those objects should have two keys: date and description. The dates should be formatted like “Nov 7, 2023”.

How does it do?

	{
	"full_name": "William Henry Gates III",
	"title": "Co-founder of Microsoft, Philanthropist",
	"events": [
	{
	"date": "Apr 4, 1975",
	"description": "Co-founds Microsoft with Paul Allen, which becomes the world's largest PC software company."
	},
	{
	"date": "Jun 27, 2008",
	"description": "Steps down from daily duties at Microsoft to focus on philanthropy with the Bill & Melinda Gates Foundation."
	},
	{
	"date": "Dec 13, 2016",
	"description": "Launches the Breakthrough Energy Coalition to invest in new clean energy technologies."
	}
	]
	}

view raw nested.json hosted with ❤ by GitHub

Nailed it 👏.

Nested JSON with a template

Another version of this is to provide it a JSON template with comments above each key with guidance about how to generate those values:

	Generate json about Bill Gates using the following template:
	{
	"full_name": "",
	// What is his current job title?
	"title": "",
	// Generate a list of 3 important events in his life
	// Each fact should have a date (formatted like "Nov 7, 2023") and a short description
	"events": []
	}

view raw described.txt hosted with ❤ by GitHub

This is more verbose, but might be work better for complex use cases where you want to provide a lot of guidance to ChatGPT about the structure and content of the output.

	{
	"full_name": "William Henry Gates III",
	"title": "Co-chair of the Bill & Melinda Gates Foundation",
	"events": [
	{
	"date": "Apr 4, 1975",
	"description": "Co-founded Microsoft, which became the world's largest PC software company."
	},
	{
	"date": "Jan 1, 2000",
	"description": "Established the Bill & Melinda Gates Foundation, focusing on philanthropy in areas like health and education."
	},
	{
	"date": "Jun 27, 2008",
	"description": "Transitioned from day-to-day operations at Microsoft to focus more on philanthropy."
	}
	]
	}

view raw nested.json hosted with ❤ by GitHub

When JSON mode doesn’t return valid JSON

Without reading the docs, it’s easy to come away believing that the API will always return valid JSON if you’re requesting the output in JSON mode. However, there is one situation when it doesn’t, and that’s if ChatGPT reaches its maximum output token limit while generating the response. From the docs:

The JSON in the message the model returns may be partial (i.e. cut off) if finish_reason is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check finish_reason before parsing the response.

Both gpt-4-1106-preview and gpt-3.5-turbo-1106 have a maximum token output of 4,096 tokens, which means if you request a large amount of data, or if you explicitly set max_tokens to a lower limit and ChatGPT hits that limit while generating the response, it will stop generating the JSON midway through it. Here’s an example to demonstrate:

	require "openai"
	unless ENV["OPENAI_SECRET_KEY"]
	puts "Please set the OPENAI_SECRET_KEY environment variable on the command line:"
	puts "$ export OPENAI_SECRET_KEY=\"sk-…\""
	return
	end
	def get_json(prompt)
	client = OpenAI::Client.new(access_token: ENV["OPENAI_SECRET_KEY"])
	response = client.chat(parameters: {
	model: "gpt-4-1106-preview",
	messages: [
	{ role: "user", content: prompt }
	],
	response_format: { type: "json_object" },
	max_tokens: 100
	})
	if response.key?("error")
	error_message = response.dig("error", "message")
	puts "Error: #{error_message}"
	return
	end
	response.dig("choices", 0, "message", "content")
	end
	prompt = "Generate JSON with a single key named facts that lists 50 facts about world history."
	json = get_json(prompt)
	puts json

view raw partial.rb hosted with ❤ by GitHub

And here’s the output:

	{
	"facts": [
	"The Great Wall of China was built over several dynasties and spans more than 13,000 miles.",
	"The Roman Empire was one of the largest empires in history, with Augustus being its first emperor in 27 BC.",
	"The Black Death, a deadly pandemic, killed up to 60% of Europe's population in the 14th century.",
	"The Mongol Empire, founded by Genghis Khan in 120

view raw partial.json hosted with ❤ by GitHub

Attempting to parse this output with something like JSON.parse will obviously not work.

Following the doc’s advice to check the finish_reason, we can update the script accordingly to return null if ChatGPT couldn’t generate the entire JSON object:

	require "openai"
	unless ENV["OPENAI_SECRET_KEY"]
	puts "Please set the OPENAI_SECRET_KEY environment variable on the command line:"
	puts "$ export OPENAI_SECRET_KEY=\"sk-…\""
	return
	end
	def get_json(prompt)
	client = OpenAI::Client.new(access_token: ENV["OPENAI_SECRET_KEY"])
	response = client.chat(parameters: {
	model: "gpt-4-1106-preview",
	messages: [
	{ role: "user", content: prompt }
	],
	response_format: { type: "json_object" },
	max_tokens: 100
	})
	if response.key?("error")
	error_message = response.dig("error", "message")
	puts "Error: #{error_message}"
	return
	end
	# If the finish reason is "length", then we ran into the max token limit
	if response.dig("choices", 0, "finish_reason") == "length"
	return
	end
	response.dig("choices", 0, "message", "content")
	end
	prompt = "Generate JSON with a single key named facts that lists 50 facts about world history."
	json = get_json(prompt)
	if json.nil?
	puts "Ran into the max token limit."
	return
	end
	puts json

view raw max_tokens.rb hosted with ❤ by GitHub

—

Hopefully this gives you an idea of what’s possible with the new JSON mode. Shout out to the OpenAI team for implementing it. And if you have any other tips about using this new feature, please drop a comment below and I’ll update this post accordingly. Cheers!