If you’ve ever tried to get structured JSON out of an LLM, you know the drill. It starts simple — you throw response_format: { type: "json_object" } at OpenAI and it mostly works. Then someone wants to try the same prompt on Claude. Then Gemini. Then DeepSeek. And suddenly you’re maintaining a switch statement the size of a CVS receipt.

We just shipped a big piece of Gatelit that handles all of this. Here’s what the problem looked like, and how we solved it.

The landscape

Let’s walk through what “structured output” actually means across the major providers. It’s not pretty.

OpenAI has the most mature implementation: response_format: { type: "json_schema", json_schema: { name: "...", strict: true, schema: {...} } }. Looks clean. But strict mode has rules — every object must set additionalProperties: false, every property must be listed in required, and numeric constraints like minimum, maximum, minLength, maxLength, and pattern are flat-out rejected. Include any of those and OpenAI throws an error.

Anthropic uses a completely different shape: output_config: { format: { type: "json_schema", schema: {...} } }. Same strict rules apply, but the wrapping is different. Also — and this is where it gets fun — Claude 3.5 and Claude 4 models don’t support native json_schema at all. They do json_object mode. You need to inject the schema into the system prompt and hope the model follows it. Only Claude 4.5+ has constrained decoding.

Google Gemini takes its own approach entirely. Schema goes under generationConfig.responseSchema, mime type is responseMimeType: "application/json", and the schema types are UPPERCASE: STRING, INTEGER, NUMBER, BOOLEAN, ARRAY, OBJECT. Does it support $ref and $defs? No — you inline them. Does it support additionalProperties? Nope. Nullable fields? Not with anyOf: [{ type: "null" }, ...] — Google wants nullable: true instead. And there’s a propertyOrdering field that only Gemini uses.

Mistral doesn’t support native json_schema at all. Neither does DeepSeek in practice — their API technically accepts it but we’ve found the output unreliable enough that we route it through prompt emulation instead. For both, the only safe path is to inject the schema as text into the system prompt, pair it with json_object mode, and cross your fingers. (At least, that’s what you’d do if you were handling it yourself.)

Cohere does its own thing: response_format: { type: "json_object", schema: {...} }. Not OpenAI-compatible, just similar enough to be confusing.

Groq mostly follows OpenAI’s API surface, but actual json_schema with constrained decoding only works on two models: gpt-oss-20b and gpt-oss-120b. Everything else falls back to json_object with system prompt injection — same trust-me-bro approach as Mistral and DeepSeek.

So: seven providers, five different request shapes, three different schema formats, and a bunch of “works on some models but not others” caveats. Multiply that by however many models your team actually uses, and it’s a full-time job just keeping up.

What we built

The core idea is simple: you write one JSON Schema, and Gatelit handles the rest. No provider-specific formatting, no provider-specific edge cases, no “oh wait this model doesn’t support that keyword.”

Under the hood, every provider gets an adapter:

applyOpenAi() normalizes your schema for OpenAI/Groq/Together/Fireworks strict mode. It walks the entire schema tree, sets additionalProperties: false on every object, auto-populates required with all property keys, and strips constraint keywords (minimum, maximum, maxLength, pattern) — moving them into the description field so the model still sees them.
applyAnthropic() maps into Anthropic’s output_config.format shape and applies the same strict-mode normalization.
applyGoogle() converts JSON Schema types to UPPERCASE, resolves $ref and $defs by inlining, converts anyOf nullable patterns to Google’s nullable: true, and drops additionalProperties since Gemini doesn’t support it.
applyMistral() and applyCohere() handle providers that do json_object + schema injection or their own variant.

The result: you send a request with a standard JSON Schema and the model name, and Gatelit produces the correct provider-specific request body. You don’t know or care that Gemini wants UPPERCASE types or that Anthropic Closes the door on minLength.

But wait — what about models that can’t guarantee valid output?

This was the part we spent the most time on. Not all models support constrained decoding. Our model catalog now tracks this per-model with a structured_output_mode:

native: Constrained decoding guarantees valid output (OpenAI, Anthropic 4.5+, Groq’s GPT-OSS models, Fireworks)
emulated: Prompt-level hints only — the model might ignore the schema (DeepSeek, most Groq models, Cohere, Mistral)
none: No structured output support

When you pick a model with emulated mode in the dashboard, you’ll see a warning badge: “This model emulates structured output via prompt hints — results may not match the schema.”

But we didn’t stop at a warning. The gateway now validates every non-streaming response against the requested schema. If the output is invalid JSON, doesn’t match the schema’s required fields, has wrong property types, or returns an array when an object was expected — Gatelit catches it and automatically retries the request (up to 2 times). If all retries fail, the request is logged with the retry count and failure reason, and you get a clear error back.

This means you can actually use DeepSeek or Mistral with structured output in production — the gateway has your back when the model drifts. The retries are tracked in the request logs so you can see which models are reliable and which aren’t.

The real win

The thing we’re most happy about: switching providers is now trivial. If you’re using structured output through Gatelit and want to move from OpenAI to Anthropic, or try Gemini, or test DeepSeek for cost — the schema stays the same. The adapter handles the translation, the validation catches failures, the retries keep things working. One schema, everywhere.

What’s next

We’re working on something even more ambitious: making every provider JSON-compatible, even ones that don’t support structured output at all. Think of it as TypeChat, but built into the gateway — enforced JSON schema parsing on Gatelit’s side, so you can get structured output from any model, regardless of what the provider supports. We’ll parse, validate, and retry until it matches, with transparent fallback. More on that soon.

If you’re tired of provider-specific schema formats, join the waitlist — structured output is live now.