Vision Understanding (Image Recognition) API

Use AI models for intelligent image analysis and understanding, supporting object recognition, scene description, text extraction, and more

APIYI provides powerful image understanding capabilities, supporting deep analysis and understanding of images using various advanced AI models. Through a unified OpenAI API format, you can easily implement image recognition, scene description, OCR text recognition, and other functions.

🔍 Intelligent Visual Analysis Supports various visual tasks including object recognition, scene understanding, text extraction, sentiment analysis, and more, enabling AI to truly “understand” images.

🌟 Core Features

• 🎯 Multi-Model Support: Top multimodal models like the Gemini 3, GPT-5, and Claude 4 series
• 📸 Flexible Input: Supports URL links and Base64 encoded images
• 🌏 Chinese Optimization: Perfect support for Chinese scene understanding and text recognition
• ⚡ Fast Response: High-performance inference with second-level results
• 💰 Cost Control: Multiple model options to meet different budget requirements

📋 Supported Vision Models

The following are current mainstream multimodal recommendations. Model IDs may change with new releases — always defer to the console.

Model Name	Model ID	Features	Recommended Scenarios
Gemini 3.1 Pro Preview ⭐	gemini-3.1-pro-preview	Strongest multimodal reasoning, rich detail	Complex image/scene analysis
Gemini 3.5 Flash 🔥	gemini-3.5-flash	Fast and low-cost, best value	Real-time recognition, batch processing
GPT-5.5 ⭐	gpt-5.5	Strong all-round vision understanding, stable	General image understanding
Claude Opus 4.7	claude-opus-4-7	Deep understanding, precise descriptions	Professional image+text analysis
Claude Sonnet 4.6	claude-sonnet-4-6	Rivals Opus, high cost-performance	Cost-effective recognition
GPT-4o	gpt-4o	Classic multimodal, mature and stable	General scenarios
Gemini 2.5 Flash	gemini-2.5-flash	Ultra-fast and cheap, GA release	Large-batch simple recognition

Most chat models now support multimodal image input: the table above lists common recommendations, not the full set. Mainstream models including the GPT-5, Gemini 3, Claude 4 series, Grok 4, Qwen, GLM, and Kimi mostly accept image input.

• 📚 Full model list and feature comparison: Popular Models (Kept Updated)
• 🔗 Live model list and pricing: APIYI Console Pricing Page (check the console for vision support)

🚀 Quick Start

1. Basic Example - Image URL

import requests

url = "https://api.apiyi.com/v1/chat/completions"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}

payload = {
    "model": "gpt-5.5",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Please describe this image in detail"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/image.jpg"
                    }
                }
            ]
        }
    ]
}

response = requests.post(url, headers=headers, json=payload)
result = response.json()
print(result['choices'][0]['message']['content'])

2. Local Image Example - Base64 Encoding

import base64
import requests

def image_to_base64(image_path):
    """Convert local image to base64 encoding"""
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

# Read local image
base64_image = image_to_base64("path/to/your/image.jpg")

url = "https://api.apiyi.com/v1/chat/completions"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}

payload = {
    "model": "gemini-3.1-pro-preview",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Analyze all text content in this image"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    }
                }
            ]
        }
    ]
}

response = requests.post(url, headers=headers, json=payload)
print(response.json()['choices'][0]['message']['content'])

3. Advanced Example - Multi-Image Comparison

import requests

url = "https://api.apiyi.com/v1/chat/completions"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}

payload = {
    "model": "gemini-3.1-pro-preview",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please compare the differences between these two images:"},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/image1.jpg"}
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/image2.jpg"}
                }
            ]
        }
    ],
    "max_tokens": 1000
}

response = requests.post(url, headers=headers, json=payload)
print(response.json()['choices'][0]['message']['content'])

4. cURL Example (Command Line)

Image URL method:

curl https://api.apiyi.com/v1/chat/completions \
  -H "Authorization: Bearer $APIYI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.1-pro-preview",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Please describe this image in detail" },
          { "type": "image_url", "image_url": { "url": "https://example.com/image.jpg" } }
        ]
      }
    ]
  }'

Local image Base64 method (encode the image to Base64, then embed it in the request body):

# 1. Convert local image to base64 (macOS / Linux)
BASE64_IMAGE=$(base64 -i path/to/your/image.jpg | tr -d '\n')

# 2. Pass the image content via a data URI
curl https://api.apiyi.com/v1/chat/completions \
  -H "Authorization: Bearer $APIYI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Analyze all text content in this image" },
          { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,'"$BASE64_IMAGE"'" } }
        ]
      }
    ]
  }'

Prefer Base64 upload for reliability: with the image URL method, the server has to download the image in real time first — if the image host responds slowly or restricts access, the download fails. Base64 embeds the image data directly in the request body, with no dependency on any external download, so it is more stable. Both methods are officially supported. Base64 is about 1.33x the size of the original image, so consider compressing large images before encoding.

5. Common Error: Image URL Download Timeout

When using the image URL method, you may receive an error like this:

{
  "error": {
    "message": "Timeout while downloading ip:port",
    "type": "invalid_request_error",
    "code": "invalid_image_url"
  }
}

This means the server timed out while downloading the image from the URL — it has nothing to do with the model, your API key, or your quota. Common causes:

1. The image host / origin server responds slowly, or is unfriendly to certain network regions
2. The image is too large and the download exceeds the time limit
3. The URL has hotlink protection, requires login, or is not a public direct link

Solutions:

• ✅ Switch to Base64 (data URI) upload (recommended, see Example 2 above) — the image data is submitted directly in the request body, completely bypassing the download step, which is the most stable option
• Use a faster, publicly accessible direct image link
• Compress the image and retry

6. Common Error: invalid base64 data (URL Mistakenly Placed in the Base64 Field)

If you receive a 400 error like the following (Claude-series wording shown here; other model series phrase it slightly differently, but the key signature is invalid base64 data):

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "...source.base64: invalid base64 data"
  },
  "request_id": "req_011CczN..."
}

It usually means an image URL was pasted into the Base64 data slot of the data URI:

// ❌ Wrong: what follows base64, is an image link, not Base64-encoded data
"image_url": {
  "url": "data:image/jpeg;base64,https://example.com/generations/temp-xxx.jpg"
}

Whatever follows the data:image/...;base64, prefix must be the Base64-encoded content of the image file itself, not an image link. The URL method and the Base64 method are two mutually exclusive ways of passing an image — they cannot be mixed. A common cause: the client code always goes through a data URI concatenation path, so remote image URLs also get concatenated in.

Correct usage, side by side:

// ✅ Image is a remote URL → pass the link directly, no prefix at all
"image_url": {
  "url": "https://example.com/generations/temp-xxx.jpg"
}

// ✅ Image is a local file / binary → Base64-encode it first, then build the data URI (see Example 2 above)
"image_url": {
  "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..."
}

Self-check: decide by the image source before sending — if the string starts with http, use the URL method; only Base64-encode and build a data URI otherwise. Also, a valid Base64 string never contains characters like :// or ? — if you see them after base64,, a link was almost certainly concatenated in.

7. Common Error: Declared Media Type Doesn’t Match the Actual Image Format

If you receive a 400 error like the following (the key signature is The image was specified using the image/png media type, but the image appears to be a image/jpeg image):

{
  "status_code": 400,
  "error": {
    "message": "InvokeModel: operation error Bedrock Runtime: InvokeModel, https response error StatusCode: 400, RequestID: e87a35ed-..., ValidationException: ...source.base64: The image was specified using the image/png media type, but the image appears to be a image/jpeg image"
  }
}

What it means: this error comes from the upstream model service’s input validation (Bedrock Runtime: InvokeModel, ValidationException in the example indicates the request reached a Claude-series upstream channel and was rejected at the parameter-validation stage). The message is remarkably literal:

• Your data URI declares the image as PNG (data:image/png;base64,...)
• But after decoding the Base64, the upstream inspected the file header (magic bytes) and found the actual content is JPEG
• Declaration and content disagree → 400. The Base64 encoding itself is fine — the media type in the prefix is what’s wrong

Common causes:

1. MIME type inferred from the file extension, but the extension lies — the file is named xxx.png but is really a JPEG with a renamed extension (download tools, chat apps, and screenshot tools all do this)
2. image/png (or image/jpeg) hardcoded in the client code, with every image getting the same prefix regardless of format
3. The image went through a processing pipeline that changed its format, but the filename stayed the same

Fix: never trust the extension — sniff the real MIME type from the file’s magic bytes before building the data URI:

import base64

def image_to_data_uri(image_path):
    """Detect the real format from magic bytes so the media type always matches the content"""
    with open(image_path, "rb") as f:
        data = f.read()

    if data[:8] == b"\x89PNG\r\n\x1a\n":
        mime = "image/png"
    elif data[:3] == b"\xff\xd8\xff":
        mime = "image/jpeg"
    elif data[:6] in (b"GIF87a", b"GIF89a"):
        mime = "image/gif"
    elif data[:4] == b"RIFF" and data[8:12] == b"WEBP":
        mime = "image/webp"
    else:
        raise ValueError(f"Unrecognized image format: {image_path}")

    return f"data:{mime};base64,{base64.b64encode(data).decode('utf-8')}"

Alternatively, re-encode with PIL — this guarantees declaration matches content in one step (and lets you compress and strip odd frames along the way):

import base64, io
from PIL import Image

def image_to_data_uri(image_path):
    img = Image.open(image_path)
    buf = io.BytesIO()
    img.convert("RGB").save(buf, format="JPEG", quality=90)
    b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"

Self-check: file xxx.png (macOS / Linux command line) reveals a file’s true format in one second; in Python, Image.open(path).format does the same. Model series differ in how strictly they validate the media type — some let mismatches through, while the Claude series (especially via the Bedrock channel) is the strictest. Write your code so the declaration always matches the content, and you’ll be safe on every model.

GPT-5 series parameter differences: if you switch the examples to a GPT-5 series model such as gpt-5.5 / gpt-5.4, note that:

1. Use max_completion_tokens instead of max_tokens
2. temperature only supports 1 (leave it at the default — do not pass other values)
3. Do not pass the top_p parameter

The Gemini and Claude series have no such restrictions and work normally with max_tokens, temperature, etc.

🎯 Common Use Cases

1. Product Recognition and Analysis

prompt = """
Please analyze this product image, including:
1. Product type and brand
2. Main features and selling points
3. Suitable target audience
4. Suggested marketing copy
"""

2. Document OCR Recognition

prompt = """
Please extract all text content from the image and organize it in the original format.
If there are tables, please present them in Markdown table format.
"""

3. Medical Imaging Assistance

prompt = """
This is a medical imaging picture, please:
1. Describe basic image information (such as imaging type, body part, etc.)
2. Label visible anatomical structures
3. Note: For reference only, not for diagnostic purposes
"""

4. Security Surveillance Analysis

prompt = """
Analyze the surveillance footage to identify:
1. Number of people and their positions in the scene
2. Any abnormal behavior
3. Environmental safety hazards
4. Timestamp information (if visible)
"""

💡 Best Practices

Image Preprocessing Recommendations

1. Format Support: Mainstream formats like JPEG, PNG, GIF, WebP
2. Size Limit: Recommended single image under 20MB
3. Resolution: Higher resolution images achieve better recognition
4. Compression: Moderate compression to improve transfer speed

Prompt Optimization

# ❌ Not Recommended: Vague prompt
prompt = "What is this"

# ✅ Recommended: Specific and clear prompt
prompt = """
Please analyze this image from the following aspects:
1. Main Objects: Identify main objects or people in the image
2. Scene Environment: Describe the shooting location and environmental features
3. Color Composition: Analyze color scheme and composition characteristics
4. Emotional Atmosphere: Emotions or atmosphere conveyed by the image
5. Possible Uses: What scenarios this image is suitable for
"""

Error Handling

import requests
from requests.exceptions import RequestException

def analyze_image_with_retry(image_url, prompt, max_retries=3):
    """Image analysis function with retry mechanism"""
    for attempt in range(max_retries):
        try:
            response = requests.post(
                "https://api.apiyi.com/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {API_KEY}",
                    "Content-Type": "application/json"
                },
                json={
                    "model": "gpt-5.5",
                    "messages": [{
                        "role": "user",
                        "content": [
                            {"type": "text", "text": prompt},
                            {"type": "image_url", "image_url": {"url": image_url}}
                        ]
                    }]
                },
                timeout=30
            )

            if response.status_code == 200:
                return response.json()
            elif response.status_code == 429:
                print(f"Rate limited, waiting to retry... (attempt {attempt + 1}/{max_retries})")
                time.sleep(2 ** attempt)  # Exponential backoff
            else:
                print(f"Error: {response.status_code} - {response.text}")

        except RequestException as e:
            print(f"Request exception: {e}")

    return None

🔧 Advanced Features

1. Streaming Output

For lengthy analyses, streaming output provides better user experience:

payload = {
    "model": "gpt-5.5",
    "messages": [...],
    "stream": True
}

response = requests.post(url, headers=headers, json=payload, stream=True)
for line in response.iter_lines():
    if line:
        print(line.decode('utf-8'))

2. Multi-turn Conversation

Maintain context for in-depth analysis:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What animal is this?"},
            {"type": "image_url", "image_url": {"url": "animal.jpg"}}
        ]
    },
    {
        "role": "assistant",
        "content": "This is a Golden Retriever."
    },
    {
        "role": "user",
        "content": [{"type": "text", "text": "How old does it look? How is its health condition?"}]
    }
]

3. Combined with Function Calling

tools = [
    {
        "type": "function",
        "function": {
            "name": "save_image_analysis",
            "description": "Save image analysis results to database",
            "parameters": {
                "type": "object",
                "properties": {
                    "objects": {"type": "array", "items": {"type": "string"}},
                    "scene": {"type": "string"},
                    "text_content": {"type": "string"}
                }
            }
        }
    }
]

payload = {
    "model": "gpt-5.5",
    "messages": messages,
    "tools": tools,
    "tool_choice": "auto"
}

📊 Performance Comparison

Model	Response Speed	Recognition Accuracy	Chinese Support	Price
Gemini 3.1 Pro Preview	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	$$
Gemini 3.5 Flash	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	$
GPT-5.5	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	$$
Claude Sonnet 4.6	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	$$
Gemini 2.5 Flash	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐	$

🚨 Important Notes

1. Privacy Protection: Do not upload images containing sensitive information
2. Compliant Usage: Follow relevant laws and regulations, do not use for illegal purposes
3. Result Verification: AI analysis results are for reference only, important decisions require manual review
4. Cost Control: Choose models reasonably to avoid unnecessary expenses

🔗 Related Resources

• Complete Code Examples
• API Pricing Information

💡 Pro Tip: Test first with cost-effective models like Gemini 3.5 Flash or Gemini 2.5 Flash, then switch to advanced models like Gemini 3.1 Pro or GPT-5.5 for production once you’ve confirmed quality. For more available models, see Popular Models or the console model list.