A few weeks ago I wrote about structured LLM extraction as a validation problem. I had barely published that explanation before replacing most of the architecture behind it.

I wasn’t planning a rewrite. I changed the model in my test notebook.

Most of Structx had been developed against 4o and older 4.5-era models. The document I normally used took about 22 seconds to process. With GPT-5.5, schema generation ran for more than a minute and then failed. After I fixed that, a consultancy agreement took almost two minutes. In another run, Structx returned a flat model full of nulls because I had given it the wrong path. The model had never seen the agreement. It had seen a string containing its filename.

Blaming the new model would have been easy. It was slower and less forgiving of some parameters, but it had also found several assumptions that 4o quietly let me get away with.

The failures kept pointing back into Structx.

• Structx was sending generation parameters that were not valid everywhere.
• Model generation carried multiple intermediate concepts that extraction did not actually use.
• Type conversion accepted too many loosely defined shapes and repaired them too late.
• Missing paths could travel surprisingly far into the pipeline.
• Async extraction was a synchronous call inside a thread executor.
• Token usage had been translated into Structx-specific summaries, losing provider details such as reasoning tokens.
• DataFrames were doing two jobs at once. They held user data and acted as an internal transport envelope full of magic columns and hidden attributes.

I started fixing these as individual bugs. Each fix exposed the next hidden contract, so the patch kept growing until very little of the old pipeline was worth preserving. Over a few days, Structx moved from 0.4.10 to 0.6.3. The production Python diff ended up at roughly 2,750 additions and 2,500 deletions.

The useful part of the rewrite wasn’t the amount of code changed. It was making the boundaries visible enough that I could tell what each stage owned, what it returned, and why it existed.

Too Many Planning Objects

The original pipeline divided planning into several concepts.

flowchart LR
    A[User query] --> B[Query refinement]
    B --> C[Extraction guide]
    C --> D[Schema request]
    D --> E[Dynamic Pydantic model]
    F[DataFrame plus hidden metadata] --> G[Data processor]
    E --> H[Extraction engine]
    G --> H
    H --> I[Result manager]
    I --> J[Flattened output and usage summary]

QueryRefinement described the expanded query, data characteristics, and structural requirements. ExtractionGuide added target columns, structural patterns, relationship rules, and organization principles. Then an ExtractionRequest described the actual model.

It looked thorough in a class diagram. Several fields never affected the extraction call at all. The model had to produce a larger planning response, the Python flow became harder to follow, and there were more places for that response to drift before extraction began.

The same pattern appeared elsewhere. DataProcessor handled normalization, batching, threads, async wrappers, and temporary files. A result manager kept mutable state while usage lived inside LLMCore. File behavior was inferred from columns such as pdf_path, multimodal, and file_type, plus DataFrame attributes such as content_sample and temporary_paths.

I could explain each abstraction on its own. I couldn’t look at the complete pipeline and quickly tell which object owned a temporary file, where usage was stored, or which planning fields actually reached the extraction prompt.

One Document Path

I started with document conversion because it was already an obvious mess.

Structx had accumulated separate readers and conversion paths for Word files, PDFs, Markdown, and other document types. We considered python-docx, Unstructured, LibreOffice, and Docling. Keeping several fallbacks sounded safe, but it meant duplicated behavior, ambiguous output quality, and a package whose dependency groups no longer explained what the code really used.

The actual requirement was narrower than “parse every document.”

Preserve the document visually and send it to a multimodal model.

That gave me one non-PDF pipeline.

flowchart LR
    A[DOCX, PPTX, HTML, text, or image] --> B[Docling]
    B --> C[Text sample for schema planning]
    B --> D[HTML export]
    D --> E[WeasyPrint]
    E --> F[Temporary PDF]
    F --> G[Multimodal extraction]
    H[Existing PDF] --> G

Existing PDFs are validated and passed through unchanged. Other supported formats go through Docling once. Its text export gives planning enough content to design a schema, while its HTML export is rendered to PDF by WeasyPrint for the actual multimodal request.

OCR and Docling’s table-structure analysis are disabled. Running OCR before a vision-capable LLM would add cost and latency to solve a problem the downstream model is already expected to solve. It was showing up as a bottleneck without providing useful output to Structx.

Docling is not lightweight. Even the slim setup needs local model dependencies and PyTorch for the formats Structx supports, so I moved the entire conversion stack behind structx[docs]. Raw strings, structured files, and existing PDFs work in the base install. DOCX, PowerPoint, OpenDocument, markup, and images opt into the heavier conversion path.

There are no legacy readers behind it and no second-best fallback. If the optional conversion dependencies are missing, Structx says so.

Fail Before Asking a Model Anything

The wrong-path notebook run exposed a more embarrassing issue.

The data argument intentionally accepts a raw string, a path, a DataFrame, or a list of dictionaries. That flexibility makes path validation less obvious because a string can be either content or a filename.

The old behavior let a missing document path look enough like raw text to enter the pipeline. It was then converted into an almost empty document, sent to the model, and returned as a plausible explanation that the agreement contained no terms. The model did exactly what the pipeline asked. The pipeline should never have asked.

Input preparation now rejects missing paths, directories, empty files, empty tabular inputs, malformed PDFs, unsupported extensions, and duplicate columns before planning begins. Strings with a supported extension or path-like shape are treated as paths and fail clearly if they do not exist.

Normalization now produces a real contract.

@dataclassclass PreparedInput:    dataframe: pd.DataFrame    pdf_rows: dict[int, PdfRow]    planning_sample: str | None    owned_paths: list[Path]

PdfRow distinguishes the original source from the PDF sent to the model. owned_paths makes temporary-file ownership visible. An existing PDF is borrowed. A PDF generated from DOCX is owned and must be deleted.

That replaces hidden DataFrame attributes and transport columns with something the type system can describe.

Structx also exposes the lifecycle when an application wants to reuse an expensive conversion.

with extractor.prepare_input(data="agreement.docx") as prepared:    schema = extractor.get_schema(        data=prepared,        query="extract agreement terms",    )    result = extractor.extract(        data=prepared,        query="extract agreement terms",        model=schema,    )

The document is converted once, can be inspected between stages, and is cleaned up when the context exits. The async context performs blocking parsing and rendering off the event loop.

Collapse Planning To What Extraction Uses

The new planning response contains exactly three things.

class ExtractionPlan(BaseModel):    instructions: str    target_columns: list[str]    extraction_schema: ExtractionRequest

One planning call turns the query and representative input into explicit instructions, selects valid input columns, and defines the smallest schema that answers the question. The rest of the pipeline consumes all three fields.

There is no separate guide model and no structural metadata that disappears before extraction.

Custom models are simpler still. If the caller already supplies a Pydantic model, Structx does not call an LLM to rediscover it. It combines the user’s query with deterministic instructions, keeps all available input columns, and starts extraction directly. Keeping every column is intentional. A cheap name-matching heuristic should not be allowed to remove evidence because a column name did not resemble a model field.

This change helped latency, but I do not treat the notebook timings as a clean benchmark. GPT-5.5 itself was slower than 4o for this workload. What the rewrite did remove was avoidable work. That included redundant planning concepts, extra custom-model planning, and type repair spread across several modules. A later run of the same document fell from nearly two minutes to about one minute, but the more important improvement was knowing where that minute went.

One Place Converts Types

Dynamic model generation had another form of implicit behavior. Type strings could arrive as Python-like annotations, JSON-ish arrays, TypeScript unions, or legacy Pydantic constrained types. Conversion logic existed in more than one module, and unknown shapes could degrade into strings rather than failing.

Now ModelField is the single validation boundary. The type system normalizes inputs such as the following.

array<string> | null  -> Optional[List[str]]PositiveInt          -> int with gt=0string[]             -> List[str]object               -> Dict[str, Any]

Ambiguous unions and unknown types are rejected. Legacy constraint names are translated to their Pydantic v2 forms. Required state, nullability, and explicit defaults are separate concepts instead of being inferred from a loose Optional string.

The model generator is now a factory rather than a metaclass. It recursively creates nested models, preserves collection types such as List, Set, FrozenSet, and Tuple, and freezes nested models when a set needs them to be hashable.

That work also made the schema useful outside one Python process. Structx now has a portable ExtractionRequest contract.

from structx import (    model_from_extraction_request,    model_to_extraction_request,)definition = model_to_extraction_request(Invoice)stored = definition.model_dump(mode="json")RestoredInvoice = model_from_extraction_request(stored)

Generated models retain their original definition. Declarative custom Pydantic models can be converted by inspecting supported fields, nested models, constraints, defaults, required state, and nullability.

The boundary is deliberately strict. Custom validators, serializers, computed fields, recursive models, and default factories contain executable behavior or cannot be represented safely as portable data. Structx rejects them instead of silently storing an incomplete schema.

There is also a versioned type-capability API for schema-builder UIs. A frontend can ask Structx which scalar types, containers, modifiers, and constraints it supports rather than copying a private Python type map.

Stop Hardcoding Model Parameters

The model switch also exposed assumptions in completion configuration.

Parameters such as temperature, token limits, verbosity, and reasoning effort are not universal. Even two models behind the same OpenAI-compatible endpoint may accept different names or value combinations. A library that hardcodes a “reasonable” default can make a valid model unusable.

Structx now has only two completion settings groups.

planning:  reasoning_effort: lowextraction:  reasoning_effort: medium

There are no implicit sampling controls or output-token limits. The user owns those decisions. LiteLLM receives drop_params=True so parameters it knows are unsupported can be removed for the selected model, while provider-specific settings still pass through.

Configuration moved from a hand-merged OmegaConf wrapper to Pydantic Settings. It can read constructor values, environment variables, .env, YAML, and file secrets with one validated precedence order. A separate planning_model lets a cheaper or faster model design schemas while the primary model handles document extraction.

LiteLLM metadata still cannot describe every custom endpoint. Structx routes the settings it was given, LiteLLM filters what it understands, and the provider remains the authority on valid values.

Async Means Async Now

The original extract_async submitted the synchronous extract method to a thread executor. That was how I found a concrete bug. The wrapper passed keyword-only arguments positionally and raised the following error.

Extractor.extract() takes 1 positional argument but 6 were given

Fixing the call made the symptom disappear, but the design was still wrong for network-bound LLM requests.

Extractor.from_litellm() now creates both synchronous and asynchronous Instructor clients around LiteLLM’s completion and acompletion. Planning, model refinement, and extraction all have native async paths. Blocking document conversion is the only work moved to a thread.

Rows remain independent requests behind an asyncio.Semaphore.

flowchart LR
    A[PreparedInput] --> B[Plan once]
    B --> C[Create row tasks]
    C --> D[Bounded semaphore]
    D --> E1[Row 1 request]
    D --> E2[Row 2 request]
    D --> E3[Row N request]
    E1 --> F[Stable ordered outcomes]
    E2 --> F
    E3 --> F

I considered packing several rows into one model call. It looks efficient until the edge cases arrive.

• prompts need token-aware packing
• every row needs a generated identity
• one oversized row can invalidate unrelated rows
• partial responses need row-level validation
• retries should repeat only failed members
• usage must still be attributed to the right input

Independent requests make rate limits more visible, isolate failures, and keep provenance exact. max_threads controls both sync workers and async in-flight requests, while batch_size limits how many tasks are scheduled at once.

Results Are Row Contracts, Not Just A Flat List

The old result surface focused on the flattened extraction output and a separate failure DataFrame. That made it hard to answer a basic question when rows could return zero, one, or several items.

Which model output came from which input row?

Every input now produces a RowResult.

@dataclass(frozen=True)class RowResult(Generic[T]):    position: int    source_index: Any    input_data: RowPayload    items: list[T]    usage: ExtractorUsage    error: str | None    @property    def status(self) -> str:        if self.error is not None:            return "failed"        return "success" if self.items else "empty"

position stays unique even if a DataFrame has duplicate index labels. source_index preserves the original label. status is derived as success, empty, or failed.

result.data remains the convenient flattened output, but result.rows is the canonical record.

for row in result.rows:    print(row.position, row.source_index, row.status)    print([item.model_dump() for item in row.items])

The failure DataFrame is now a derived view rather than a second mutable source of truth. Counts are explicit too. Attempted rows, successful rows, empty rows, failed rows, and extracted output are not assumed to be the same number.

Keep Provider Usage Instead Of Reinventing It

Usage tracking went through a similar simplification.

Structx used to translate provider usage into its own models and expose helper methods around them. That abstraction hid details. LiteLLM was returning completion_tokens_details.reasoning_tokens, but Structx showed no thinking tokens because the information was lost or read from the wrong level.

The new tracker stores the raw provider usage object under one of two real model-backed steps.

• schema_generation
• extraction

Computed totals understand common input/output, reasoning/thinking, and cache field names without replacing the original object. result.usage contains the whole operation. Every RowResult contains usage for its own extraction call.

This makes cost attribution possible without inventing another token standard.

print(result.usage.total_tokens)for row in result.rows:    print(row.source_index, row.usage.total_tokens)

It also fixed a smaller API annoyance. Enum keys serialize as ordinary strings in both model_dump() and JSON output.

The Architecture After The Rewrite

The current flow is smaller even though the library can do more.

flowchart LR
    A[Raw input] --> B[InputProcessor]
    B --> C[PreparedInput]

    C --> D{Custom model?}
    D -->|no| E[One compact planning call]
    E --> F[ExtractionRequest]
    F --> G[ModelGenerator]
    D -->|yes| H[Deterministic instructions]

    G --> I[ExtractionStrategy]
    H --> I
    I --> J[BatchProcessor]

    J --> K1[Independent text row]
    J --> K2[Independent PDF row]
    K1 --> L[ExtractionEngine]
    K2 --> L
    L --> M[RowResult]
    M --> N[ResultCollector]
    N --> O[ExtractionResult]

    O --> P[result.data]
    O --> Q[result.rows]
    O --> R[result.usage]

The main boundaries now have one job.

• InputProcessor validates and normalizes ownership.
• ModelOperations creates a compact plan or deterministic custom-model instructions.
• ModelGenerator turns portable definitions into Pydantic models.
• BatchProcessor schedules independent rows with bounded concurrency.
• ExtractionEngine builds text or PDF messages and calls LLMCore.
• LLMCore owns provider calls, transient retries, and raw usage capture.
• ResultCollector projects ordered row outcomes into list or DataFrame views.

That list is not dramatically shorter than the old one. The difference is that data no longer changes meaning between modules. There is a typed object at each boundary, and ownership travels with it.

Tests Changed How I Worked On The Library

Before this pass, the repository had no tracked test modules. That made every refactor depend too heavily on the notebook and whichever model I happened to be using.

The new suite separates deterministic behavior from expensive external checks.

• unit tests cover type normalization, schema conversion, input validation, result collection, configuration, usage, retries, and row scheduling
• integration tests convert real example DOCX and PDF files
• live tests use configured endpoint variables to exercise dynamic planning and native async extraction
• document and live suites are opt-in, so the ordinary test run stays fast

The current default run is 99 passing tests with five opt-in tests skipped. More importantly, most architecture work can now be verified without spending tokens or waiting for a document model.

What I Am Keeping From This Rewrite

The first lesson is to change models earlier. An abstraction can look provider-neutral while one familiar model quietly accepts every assumption. Switching models turned out to be an architecture test, not only a quality comparison.

I am also much more suspicious of intermediate models now. If one stage generates a field and the next stage ignores it, that field is probably prompt decoration. ExtractionGuide looked useful because it had a good name and a lot of detail. Removing it made the pipeline easier to explain and did not take anything away from extraction.

The same applies to resource ownership. Temporary files hidden in DataFrame metadata worked until cleanup, reuse, and async execution mattered. Borrowed and owned paths needed to be different values in the program, not facts I had to remember while debugging it.

Some choices remain deliberately boring. Structx keeps LiteLLM’s provider usage instead of translating it into a supposedly universal format. Docling stays behind a heavy optional dependency instead of being disguised by several weak fallbacks. Rows stay as independent requests until batching can preserve their identity, failures, retries, and usage correctly.

Where Structx Ended Up

The public call still looks almost the same.

result = extractor.extract(    data="agreement.docx",    query="extract obligations, deliverables, and payment terms",)

But nearly everything below it is different.

The document path is singular instead of fallback-heavy. Input ownership is explicit. Planning returns only what extraction consumes. Model types have one normalization boundary. Schemas can leave the process and come back. Async calls are truly async. Results preserve provenance. Usage preserves provider detail. Configuration does not guess what a model supports.

The rewrite did add portable schemas, reusable prepared input, native async calls, and row-level usage. None of those started as feature requests. They became possible once Structx stopped passing loosely defined state between stages.

The next model change will probably expose something else. At least now I have tests for the deterministic parts and typed boundaries around the expensive ones. I would rather debug that than another DataFrame with secret attributes.