I have been thinking about structured LLM extraction as a validation problem. The tempting version of the problem is simple. Give an LLM some text, ask for JSON, parse the JSON, move on. That works just long enough to feel convincing. Then a field changes shape, a nested object comes back as a string, a date is half-normalized, a table row disappears, or the model invents a key that the downstream code has never heard of.

At that point the schema becomes the contract the prompt was only hinting at.

That is the idea behind structx, a Python library I built for extracting structured data from text and documents using LLMs. I have a project page about structx that explains the library from the outside. This post is the internal version: how a natural-language extraction request becomes a typed Pydantic model, how Instructor enforces that model at generation time, and why the document path uses multimodal PDF processing for layout-sensitive files.

The design goal is simple: turn user intent into a validation contract, run the model against that contract, and keep enough failure and usage information around to improve the system later.

Why JSON Is Not Enough

Raw JSON is a serialization format. On its own, it is a weak boundary for an LLM-powered application. If an extraction pipeline only asks for JSON, the application still has to answer a bunch of questions after the model responds:

  • Did the model include every required field?
  • Did it preserve nested structure?
  • Did it use the expected types?
  • Are invalid values caught before they enter the database?
  • Can failures be retried or inspected?
  • Can a downstream service trust the output shape?

The more production-facing the workflow becomes, the less satisfying “the model usually returns valid JSON” feels. Structured extraction needs a boundary where untrusted model output becomes typed application data.

In structx, that boundary is Pydantic.

The model is allowed to reason over messy text or visual documents. The application is allowed to accept only a typed object that passes validation. That separation is the whole design.

The Pipeline Shape

The public API tries to stay simple:

from structx import Extractorextractor = Extractor.from_litellm(    model="gpt-4o",    api_key="your-api-key",)result = extractor.extract(    data="System check on 2024-01-15 detected high CPU usage on server-01.",    query="extract incident date, affected system, and event details",)

Underneath that call, there are several steps:

flowchart LR
    A[User query] --> B[Query refinement]
    C[Input data or document] --> D[Content sample / file processor]
    B --> E[Extraction guide]
    D --> F[Schema generation]
    E --> F
    F --> G[Dynamic Pydantic model]
    G --> H[Instructor structured extraction]
    H --> I[Validated result objects]
    H --> J[Failed rows / retries / token usage]

That flow matters because extraction starts before the final extraction call. The library first asks the model to help define what kind of structure is needed, then turns that structure into a real Python type, then uses that type as the response contract.

From Textual Intent To A Schema

The extraction starts with a natural-language query. That query gets refined into a QueryRefinement object. The point is to make the user’s request more explicit before schema generation. The refined object captures the expanded request, data characteristics, and structural requirements.

Then structx generates an ExtractionGuide. That guide identifies which input columns or content regions matter, along with structural patterns and relationship rules. For tabular input this helps focus extraction on the right columns. For document input it keeps the schema generation step anchored to the actual content instead of letting the model invent fields in the abstract.

The schema itself is represented as an ExtractionRequest:

class ExtractionRequest(BaseModel):    model_name: str    model_description: str    fields: list[ModelField]

Each ModelField includes a name, type, description, optional validation rules, and optional nested fields. That is the bridge between language and code. The LLM can propose a structure, but it has to propose it through a typed schema description that the library can convert into an actual Pydantic model.

Dynamic Pydantic Models

The most important internal piece is the dynamic model generator. structx uses pydantic.create_model to build runtime models from the generated ExtractionRequest. It supports nested models, lists, dictionaries, optional fields, type aliases, and validation metadata. At a high level, the model generator does this:

  1. Normalize model field types.
  2. Validate type strings against a safe type map.
  3. Recursively create nested models when fields contain nested_fields.
  4. Attach field descriptions and validation rules.
  5. Register generated models so nested types can reference each other.
  6. Return a concrete Pydantic model class.

This is where the validation contract becomes real. The output becomes a Python model with a schema, field types, descriptions, optional validation constraints, and nested structure. That also makes the system inspectable. The generated model can be printed, reused, refined, or passed explicitly into a later extraction call.

result = extractor.extract(    data="incident.log",    query="extract incidents with timestamps, systems, severity, and details",)print(result.model.model_json_schema())

The query generated the schema, and the schema then becomes part of the application instead of staying buried inside the prompt.

Instructor As The Enforcement Layer

Pydantic defines the contract. Instructor is the mechanism that asks the LLM to produce output conforming to that contract. In the text path, structx wraps the extraction model inside a container model:

container_model = create_model(    f"{extraction_model.__name__}Container",    __base__=BaseModel,    items=(list[extraction_model], Field(description="Extracted items")),)

That wrapper exists because extraction usually returns a list of items, and the library also needs enough access to the raw completion metadata to track usage. The LLM call is made through Instructor with the container model as the response model. If the response cannot satisfy the schema, it is treated as an extraction failure before it can become questionable application data.

That gives the extraction step a stronger boundary than a plain “please output JSON” instruction. It is a small architectural distinction, but it changes how I think about LLM features. The model can be probabilistic. The boundary around the model should be explicit.

Why PDFs Are A Separate Path

Documents are where the extraction problem gets more annoying. If the input is a plain text string, chunking is often fine. If the input is a PDF invoice, contract, medical form, or table-heavy report, chunking can destroy the very context that makes the document understandable.

The failure modes are familiar:

  • columns collapse into the wrong order
  • table rows lose their relationship to headers
  • footnotes and section labels appear in strange places
  • values split across chunks lose surrounding context
  • visual grouping disappears

So structx treats unstructured documents differently. For PDFs, it uses Instructor’s multimodal support directly. For Word and text-like files, the default path converts the content to a PDF representation first, then sends that rendered document through the multimodal extraction path.

flowchart LR
    A[PDF] --> D[Multimodal extraction]
    B[DOCX] --> C[Docling to Markdown]
    C --> E[Markdown to PDF]
    E --> D
    F[Text or Markdown] --> G[Styled PDF]
    G --> D

That sounds indirect until you think about what the model sees. A PDF page preserves layout, tables, headings, and visual grouping. For many extraction tasks, the rendered page is a better source than extracted text.

There are still fallback modes. The library can read text chunks or extract PDF text when needed. But the default document strategy is to preserve layout as long as possible.

Failure Is Part Of The Interface

One thing I wanted to avoid was pretending extraction always works. structx returns an ExtractionResult that contains successful data, failed rows, the generated model, and token usage. That shape is deliberate. The caller should be able to inspect successes and failures together.

result = extractor.extract(    data="reports/",    query="extract incident summaries and affected systems",)print(result.success_count)print(result.failure_count)print(result.success_rate)print(result.failed)

Retries are handled with tenacity, using exponential backoff around extraction calls. That is useful for transient API failures and occasional model failures, but the retry loop has a narrow job. It can improve robustness around temporary failures. It cannot replace evals, better schemas, or better prompts. Returning failures keeps that distinction visible.

Token Usage Is Also A Signal

The library tracks token usage per step:

usage = result.get_token_usage()if usage:    print(usage.total_tokens)    for step in usage.steps:        print(step.name, step.tokens)

This matters because cost and quality are tied together in LLM systems. A schema generation step, a multimodal document extraction, and a retry-heavy run have different cost profiles. If the system is going to be used in a real pipeline, the caller needs to see where tokens are going.

Token usage is only one piece of observability. The next version of this kind of system should connect token usage, latency, validation failures, and extraction quality into a more explicit evaluation loop.

What Validation Does Not Solve

Validation gives the system a shape boundary. Pydantic can tell me that a field is a string, a date, a list, or a nested object. It can enforce shape. It can catch missing or malformed values. It can make the response usable by downstream code. Correctness needs another layer.

That distinction matters. A wrong invoice total can still be a valid decimal. A wrong party name can still be a valid string. A hallucinated clause can still fit a perfectly reasonable schema. The stack I want around this kind of system looks more like this:

flowchart LR
    A[Schema validation] --> B[Shape is acceptable]
    B --> C[Field-level checks]
    C --> D[Business rules]
    D --> E[Eval set comparison]
    E --> F[Production monitoring]

The schema prevents malformed data from moving forward. Evaluation tells you whether the system is actually good.

That is the part I would add next if I were turning structx into a more complete LLMOps project: a regression dataset, structured extraction metrics, trace-level observability, and comparisons across prompt-only, generated schema, custom schema, and multimodal paths.

The Tradeoffs

This design has tradeoffs. Dynamic schemas are powerful, and they can be wrong. If the generated model misunderstands the user’s request, the rest of the pipeline may faithfully extract into the wrong shape.

  • Multimodal document extraction preserves layout, and it can cost more than text extraction. It also depends on model support for PDF or image-like inputs.

  • Retries improve resilience, and they can hide repeated semantic failures if you do not inspect the failed rows.

  • Generated models make exploration faster, and production systems may still want approved schemas for critical workflows.

Those tradeoffs are part of the design. LLM systems need explicit boundaries because the model is only one part of the system.

Possible Next Features

The next natural step is to make validation more explicit. Right now the generated Pydantic model gives structx a strong schema boundary: field names, field types, nested structure, optional constraints, and parsing behavior. That boundary can grow into value-level validation without inventing a new validation framework.

Pydantic already has the right primitives for many deterministic checks:

  • field constraints such as length, range, pattern, and allowed values
  • nested validation for generated submodels
  • model-level checks for cross-field consistency
  • custom validators for domain rules

For a document extraction task, that could mean checking that a total is positive, a date is plausible, a currency is in an allowed set, or a line-item sum agrees with the extracted invoice total. Those are ordinary software rules, and they belong close to the schema.

There is also a second class of validation that needs context from the source. For example, an extracted vendor name might have the right type and still be the wrong value. A future version could add semantic checks for whether selected fields are grounded in the source document, whether a summary is faithful to the input, or whether a value should be routed to human review.

That is where RAG and LLM evaluation tooling starts to become useful. A library like Ragas is usually discussed around retrieval workflows, but the same general idea applies here: separate shape validation from quality validation, and make the quality checks visible instead of burying them in the prompt.

The interface I would want is something that lets a caller attach validation at the extraction boundary:

result = extractor.extract(    data="invoice.pdf",    query="extract invoice number, vendor, subtotal, tax, total, and line items",    validation={        "invoice_number": {"min_length": 3},        "line_items[].amount": {"ge": 0},        "total": {"ge": 0},        "$semantic": [            {                "type": "grounded_in_source",                "fields": ["invoice_number", "vendor", "total"],                "severity": "warning",            }        ],    },)

The result could then carry a validation report next to the extracted data, failed rows, generated model, and token usage. That would make it easier to separate several different states:

  • structurally valid
  • structurally invalid
  • structurally valid with warnings
  • semantically uncertain
  • ready for downstream use
  • needs human review

Observability fits into the same direction. The existing token usage tracking is a useful start, but a fuller system should trace each extraction stage, the generated schema, validation failures, retry behavior, and quality checks. That would turn structx from a structured extraction utility into a small LLMOps surface for document intelligence workflows.

The Bigger Lesson

The part of structx I care about most is the pattern:

  1. Start with a natural-language extraction goal.
  2. Turn it into an explicit schema.
  3. Convert the schema into a runtime validation model.
  4. Use that model as the generation contract.
  5. Preserve document layout when layout matters.
  6. Return failures and usage so the caller can inspect model behavior.

That is the pattern I keep coming back to in LLM application work. The prompt is important, but the application boundary is more important. A useful LLM system needs to know what it will accept, what it will reject, what it will retry, and what it will measure. That is what structured validation gives you: a place to stand when the model itself is allowed to be flexible.

structx is available on PyPI, with source on GitHub and documentation at structx.aolabs.dev.