Multimodal AI: How to Work with Vision + Text (2026)
Send images with a precise question, verify extracted details, and budget for image processing without confusing a model’s description with reliable evidence.
The short answer
Choose a current image-capable model and send only necessary, permitted images with a clear text task. Validate its output against the original. Legacy GPT-4 is text-only; image support belongs to specific models and endpoints, not a brand label.
Send images with a precise question, verify extracted details, and budget for image processing without confusing a model’s description with reliable evidence.
01 / The story
Update the “GPT-4 Vision” shorthand
Multimodal input combines types such as images and text. “GPT-4 Vision” is a historical shorthand, not a universal current endpoint. OpenAI’s GPT-4 model page marks images unsupported. Use the current image-input guide and model catalog to select an actual image-capable option.
Claude’s checked vision documentation accepts images as inputs to its Messages API. Google documents image understanding for supported Gemini models. Image understanding is not the same as image generation, and support for images does not automatically establish video or audio support.
02 / The story
Prepare an image you have permission to send
Use a legible crop of the relevant content, with enough context to avoid changing its meaning. Remove private details that the task does not require. Readable text, sensible orientation, and clear source labels help a reviewer and may reduce unnecessary processing.
Do not remove forensic or ownership context when that matters. Avoid sending identity documents, medical images, or customer records without appropriate authorization and provider terms. Verify image formats, request-size limits, and model-specific resolution behavior in current documentation.
03 / The story
Make a first Claude vision request
The checked Claude guide supports an image content block with base64 data, followed by a text question. The example reads a local JPEG named sample.jpg that you must supply with permission. It demonstrates request structure, not image accuracy. Confirm the file, model availability, SDK, and billing before execution.
Ask for observed details and an explicit “unclear” when the image does not support an answer. A model can still misread an object or letter. Do not treat a short answer as verified just because you requested uncertainty.
python / Example
import base64
from pathlib import Path
import anthropic
image_data = base64.b64encode(Path("sample.jpg").read_bytes()).decode("ascii")
client = anthropic.Anthropic()
message = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=256,
messages=[{
"role": "user",
"content": [
{"type": "image", "source": {
"type": "base64",
"media_type": "image/jpeg",
"data": image_data
}},
{"type": "text", "text": "Describe visible objects briefly. If a detail is unclear, say unclear."}
]
}]
)
for block in message.content:
if block.type == "text":
print(block.text)04 / The story
Use each provider’s actual image format
OpenAI’s image-input guide documents endpoint-specific ways to supply images, including URLs or encoded data in supported interfaces. Gemini’s image-understanding guide documents inline image data and file-based inputs. Do not paste a Claude image block into another API and expect compatibility.
Keep image loading, validation, and provider-specific encoding inside a checked adapter. If users supply URLs, do not let your server fetch arbitrary internal addresses; allowlist appropriate sources and impose size and time limits. If you use base64, it is encoding, not encryption.
05 / The story
Choose checkable use cases and define failures
Start with draft alt text, extraction from a permitted invoice image, or a summary of a chart whose source figures are available. Show the original to the reviewer. For invoice amounts, verify arithmetic and exact fields before creating a record, and require approval before payment.
Small print, glare, rotation, overlapping objects, spatial relationships, and ambiguous charts can cause mistakes. Image descriptions are not identity verification, medical diagnosis, or proof that a photographed claim is true. High-impact uses need expert evaluation beyond a tutorial.
| Use case | Useful check | Important limit |
|---|---|---|
| Draft alt text | Compare objects and purpose with the image | Do not invent identity or intent |
| Invoice extraction | Verify exact amounts and source fields | Never authorize payment solely from generated output |
| Chart summary | Check original labels and data | Visual interpretation can miss scale or units |
| Screenshot support | Confirm the visible message and steps | Hide tokens, private details, and customer data |
06 / The story
Measure accuracy and image costs separately
Create a labeled set of images resembling your real inputs and score exact fields, omitted details, and correct “unknown” answers. Keep a separate set for later evaluation. Qlauson has not run a comparative vision benchmark, so no provider accuracy percentage is claimed.
Image costs can depend on resolution, tiles, tokenization, model, endpoint, and context tier; a file’s byte size is not a reliable bill estimator. Inspect returned usage and current pricing. Include text output, reasoning where billed, and any tools. Recheck the published Gemini 2027 rate schedule and do not assume new vision capabilities or accuracy gains will arrive next year.
A closer look
Evidence, limits, and the 2027 outlook
The sample is documentation-based and has not been executed against a paid API. Qlauson makes no medical, identification, or benchmark claim. Supply your own permitted JPEG for the code example; it is not used as site artwork.
Source check: October 12, 2026. These are documentation-based guides, not Qlauson performance tests. API access, limits, model IDs, licenses, and prices can change; confirm the linked primary sources for your account and region before spending money or sharing data. Code samples are illustrative and have not been run against paid provider APIs.
2027 outlook: Do not treat an unannounced release, better accuracy, faster response, lower price, or continued model availability as a fact. Only an explicit provider announcement can establish a planned change, and announced plans can still change. Recheck the source before a 2027 budget or deployment.
Put it to work
Before you put this into practice
Use these checks for a small, non-sensitive trial; this is not a product score.
0 of 6 considered
These questions are editorial guidance, not a product score or verified assessment.
Keep learning
Further reading
- How to Use Claude API: Complete Guide for Developers
Qlauson · APIs
- Claude vs ChatGPT vs Gemini: Which AI to Use (2026)
Qlauson · AI Tools
- Anthropic vision guide
External resource · Image content blocks, source formats, and limitations. Opens in a new tab.
- OpenAI image and vision guide
Primary documentation · Supported image inputs, endpoint formats, and token costs. Checked Oct 12, 2026. Opens in a new tab.
- Google image understanding
Primary documentation · Inline/file image input patterns. Checked Oct 12, 2026. Opens in a new tab.
- OpenAI GPT-4 model card
Primary documentation · Text-only legacy model distinction. Checked Oct 12, 2026. Opens in a new tab.
- Google Gemini pricing
Primary documentation · Input categories and announced rate changes. Checked Oct 12, 2026. Opens in a new tab.
Read next
ChatGPT and Your Browser: What Can AI Actually See, Read and Do?
What can ChatGPT see in your browser? Understand open-tab access, official extensions, cloud browsing, website permissions and the privacy risks before sharing.
Continue readingNewsletter
Get new Qlauson stories by email.
An occasional email when we publish a new guide or update an important one. No unrelated promotions, and you can unsubscribe from any email.
