Hybrid LLM

From Simple Wiki

The Hybrid LLM OCR engine pairs a conventional OCR engine with a large language model (LLM) such as Claude, ChatGPT or Gemini. The OCR engine reads the page quickly and cheaply, and the LLM is called only for the index field values that Zone OCR, template and dictionary matching could not find. The result is the accuracy of an AI model on handwriting, forms and irregular layouts, at a fraction of the cost and time of sending every field of every page to the model.

Hybrid LLM is a choice on the OCR Options tab of the Job Settings Wizard, alongside Standard, Professional, SimpleOCR and Cloud OCR.

How It Works[edit | edit source]

"Hybrid LLM" is really two engines working together: one produces the page text, and the other reads the index fields out of it.

  1. SimpleIndex first runs the Text Engine you choose (Standard, Professional, SimpleOCR or Cloud OCR) to read the page. This is a normal OCR pass, and the same one a job without an LLM would make.
  2. Your Zone OCR, template and dictionary index fields are filled from that text as usual.
  3. For any OCR field that is still blank, SimpleIndex asks the LLM to find the value. Only the missing fields are requested, not the whole field list.
  4. If every field was resolved by OCR, the LLM is not called at all and nothing is charged for that document.

The Text Engine's output is still what is used for Full-Page OCR text files and searchable PDFs, so those do not change when you add an LLM.

Setting Up a Hybrid LLM Job[edit | edit source]

  1. On the OCR Options tab of the Job Settings Wizard, set the OCR Engine to Hybrid LLM.
  2. Click the settings button to open the LLM Extraction Settings window.
  3. Choose the LLM Provider, Model and enter the API Key (see below).
  4. Choose the Text Engine and Analysis Mode.
  5. Optionally edit the Extraction Prompt, then click OK.

LLM Provider, Model and API Key[edit | edit source]

Provider Notes
Claude Anthropic's hosted API.
OpenAI OpenAI's hosted API (ChatGPT models).
Gemini Google's hosted API.
Perplexity Perplexity's hosted API.
Azure An Azure OpenAI Service deployment. Enter the full deployment URL from the Azure AI Foundry portal in the Endpoint field.
DeepSeek DeepSeek's own hosted API, or a local DeepSeek vision model. The endpoint is filled in from the Model you select.
Custom Any OpenAI-compatible endpoint: a local Ollama, LM Studio or vLLM server, or a hosted service such as Groq or Together.ai. Enter the chat-completions URL in the Endpoint field, for example http://localhost:11434/v1/chat/completions for a local Ollama install.

Select a model from the list, or type the name of one that is not listed. Click Get Key... to open the provider's page for creating an API key. The key is stored encrypted and is only valid on the computer that saved it.

For Claude and Gemini you can also choose a Skill/Context File - a local .md, .skill or .zip file with extra instructions or reference data that helps tune the extraction. Other providers do not accept file attachments, so the box is disabled.

Use the Settings Configured for This Computer[edit | edit source]

Check Use the settings configured for this computer to take the provider, endpoint, model and API key from the defaults set up for the computer rather than saving them in the job. They are set once in the Global Settings Wizard.

This keeps the API key out of the job file, so a job can be copied to another computer, or handed to an end-user, and will use whatever account that computer has. Everything else - the prompt, the Text Engine, the Analysis Mode and translation - stays with the job, because those describe the documents and not the account. If a provider or key is entered in the job itself, that takes precedence.

Text Engine[edit | edit source]

The Text Engine reads the page before the LLM is asked anything. Choose the one that suits your documents:

  • Standard - the built-in Tesseract engine. No extra cost.
  • Professional - FineReader, for the best local accuracy. Requires a Professional license.
  • SimpleOCR - the lightweight engine.
  • Cloud OCR - a cloud OCR service as the first pass. Click Configure... to choose the provider and model and enter its credentials. Only modes that return a plain text transcript can be used here; the Forms, Invoice and Tables modes are not text sources. See Cloud OCR.

Analysis Mode[edit | edit source]

The Analysis Mode controls what the LLM is shown. Hybrid is the default.

Mode The LLM receives Best for
OcrOnly The OCR text and where it sits on the page. No images are sent. Clean printed documents. Lowest cost, because text uses far fewer tokens than images.
VisionOnly The page images only. No local OCR pass is made. Documents where local OCR is of little use, such as handwriting.
Hybrid Both the OCR text and the page images. Mixed documents - printed forms with handwritten entries, checkboxes and marks - where the text gives accuracy on the printed parts and the image lets the model see the rest.

VisionOnly and Hybrid send the page images to the model, so they are disabled when the provider and model you chose cannot accept images (for example DeepSeek's hosted text-only models). If you pick an image mode and then choose such a model, SimpleIndex switches the model to one that can see the page rather than let the job fail at the provider.

With OcrOnly and Hybrid the LLM is asked only for the fields OCR left blank. With VisionOnly there is no local pass to fall back on, so the model reads the page directly in a single call.

Index Fields[edit | edit source]

OCR Fields[edit | edit source]

Set up Zone OCR, template and dictionary fields as you would for any job. These are tried first. The LLM is asked only for those that came back blank, and it finds each one by its field name, so give fields descriptive names such as Invoice Number or Ship To Address.

Registration Field[edit | edit source]

With OcrOnly and Hybrid, a job can name a field that local OCR must have found for the page to count as a real match. If that field is still blank, SimpleIndex assumes the page is not one the job's fields apply to and skips the LLM for it, so you are not charged for asking a model to fill in fields on a page it was never meant to read.

Checkbox and Multiple-Choice Fields[edit | edit source]

OMR fields (checkboxes and multiple-choice answers) are answered by the LLM, which needs to see the page. They are therefore available in Hybrid and VisionOnly modes and are always requested, not only when blank. Use the name of the OMR field as the question as it appears on the document, for example Question 3. The value saved is the label of the option that was selected, such as C, rather than a true/false value.

Table Fields[edit | edit source]

Table-type index fields can be filled by the LLM as well. The field's template picks out the table it describes from whatever tables the model finds on the page, in the same column count,header text form used with the AWS table engines.

Extraction Prompt[edit | edit source]

The Extraction Prompt is added to the request that SimpleIndex builds from your field list. Use it for guidance that applies to every document in the job - the document type, date formats, how to treat a blank value, or business rules - and leave the field list itself to the index fields.

Check Return raw model output to save the model's answer exactly as it comes back, without parsing it into field values. This is for custom prompts that ask for a different format, such as XML or a different JSON layout.

Translation[edit | edit source]

Enter a language in Translate To to have the document translated. The translation keeps every line where it was on the page. Labels, headings and descriptions are translated; numbers, dates, codes and other data values are copied across unchanged.

By default the translation takes the place of the original document in the batch. Check Keep Original to keep both, for example for customs or contract files where the original is the record and the translation is a reading aid.

Translation needs the OcrOnly or Hybrid Analysis Mode, because it depends on knowing where the text sits on the page.

Sample Job[edit | edit source]

The installed AI and LLM sample folder contains a Patient Intake Form job that reads handwritten entries and checkboxes with an LLM. See Sample Jobs.

Privacy and Pricing[edit | edit source]

IMPORTANT Documents are sent to the LLM provider you choose, so check that the provider's terms suit the data in your documents. A local model reached through the Custom provider keeps documents on your own network.

Your API key is linked to your own account with the provider. Usage fees are charged by the provider according to its price list, are paid directly to the provider, and are separate from your SimpleIndex license. Fees are based on tokens, and a page image uses many more tokens than the text of the same page. To keep costs down:

  • Use OcrOnly for clean printed documents.
  • Make good use of zone, template and dictionary fields so that the LLM only has to be asked for what OCR cannot find.
  • Set a registration field so that unrelated pages are skipped.
  • Choose a smaller, faster model where accuracy allows.

Troubleshooting[edit | edit source]

Turn on Debug Mode and review the process log. For each document it records the provider and model used, the Analysis Mode and Text Engine, the fields the model was asked for and the fields it returned. A field that was asked for and came back empty means the model could not find it; a field that was never asked for was already filled by OCR, or was skipped by the registration field.

Related Pages[edit | edit source]