How to Integrate Google Gemini API: Pro, Flash, and Vertex AI Models

As a powerful ChatGPT alternative, Gemini offers multimodal model support for video processing and more. When integrating the Gemini API into production, we encountered a typical problem: the 1 million token context window handled large documents, but p99 latency reached 12 seconds. The reason was the lack of chunking and incorrect generation_config settings. Our client from fintech wanted to process quarterly reports (up to 500 pages) in real time. Without optimization, the model produced responses in 10–15 seconds, which was unacceptable for a chat.

A common mistake when integrating is using the model without chunking on long documents: tokens fill the context, making the response slow or incomplete. We developed a chunking strategy with a 200-token overlap, reducing p99 to 2 seconds on documents up to 1 million tokens. Additionally, incorrect safety settings can block legitimate requests — we configure thresholds for each scenario to avoid losing important data. Unlike GPT-4o, Gemini provides more context (1M vs 128K) but requires precise tuning of generation_config. For analytical tasks, we set temperature 0.2; for creative, 0.9. Gemini also natively handles video and audio, opening scenarios for media platforms.

Solving Problems with Gemini API Integration

High latency on long contexts

Without chunking and streaming, response time grows linearly. We split documents into 2000-token chunks with a 200-token overlap. Streaming with stream=True reduces perceived latency by 3x.

Unstable JSON response

The model sometimes returns invalid JSON if response_mime_type is not specified. We force application/json and add a fallback check.

Noisy results from multimodal requests

Low-quality images or video with artifacts. We normalize input: compress images to 1024×1024, convert video to H.264.

How we do it: a case with RAG and Function Calling

In our practice, there was a project for a fintech startup: an assistant analyzes company reports and answers questions in real time. Stack: Gemini 1.5 Pro, pgvector for embeddings (1536-dim), LangChain for orchestration. To get accurate numeric answers, we added Function Calling — the model calls get_financial_metric(ticker, period) and returns data from the database.

import google.generativeai as genai

def get_financial_metric(ticker: str, period: str) -> dict:
    # query SQL database
    return {"ticker": ticker, "revenue": 125000000, "currency": "EUR"}

model = genai.GenerativeModel("gemini-1.5-pro", tools=[get_financial_metric])
response = model.generate_content("Apple (AAPL) profit for the last reported quarter")
print(response.text)  # (metric value)

Without Function Calling, the model could hallucinate numbers. With the tool, accuracy is guaranteed.

Work process

Estimated timelines

• Basic integration of one endpoint — from 1 day.
• Scenario with RAG and Function Calling — from 3 days.
• Full production with Vertex AI and CI/CD — up to a week.

Cost is calculated individually — contact us, we'll evaluate your project within 24 hours. Basic integration starts at $500, while advanced scenarios with RAG and function calling begin at $1,500. Typical savings: by using Gemini Flash for simple queries, you can cut costs by up to 40% compared to Pro, often saving $500–$2,000 per month depending on volume.

What's included

• Integration code with Gemini API (Python / TypeScript).
• Documentation for endpoints and content generation.
• Access and IAM configuration (Google Cloud Service Account).
• Team training (2 hours online).
• Post-deployment support — 2 weeks.

Model comparison: GPT-4o vs Gemini Pro vs Flash

Gemini Flash is 5x faster than Pro in p50 latency. Flash is the number one choice for chats where speed matters. Pro is for deep document analysis. Flash is more cost-efficient than Pro at high volumes.

Optimizing p99 latency

We use aiohttp connection pools, enable streaming, and set max_output_tokens = 1024 for short responses. For Flash model, p99 latency stays under 1 second with 50 parallel requests. Streaming setup:

1. Install google-generativeai>=0.3.0.
2. Initialize the model: genai.GenerativeModel("gemini-1.5-flash").
3. Set generation_config with max_output_tokens=1024 and temperature=0.2.
4. Call model.generate_content(..., stream=True).
5. Process chunks in a loop for chunk in response:.

Common integration mistakes: safety settings not configured (model blocks valid requests), no fallback (client crashes on network error), ignoring rate limits (free tier has 60 RPM, in production switch to paid tier or Vertex AI).

Why Vertex AI for enterprise?

Vertex AI provides IAM, VPC-SC, auditing, and SLA 99.9%, along with MLOps features like model registry and monitoring. Our team has extensive experience in ML integration across various industries. We have Google Cloud certifications and experience deploying in europe-west4 and us-central1 regions. With extensive experience in AI development, we have implemented over 50 integrations with Google AI, including major fintech and e-commerce projects.

Reducing API costs

Proper model selection and configuration reduce costs. For simple requests, use Flash; for complex, Pro. Caching embeddings reduces the number of API calls. For more on cost reduction, see Google AI Documentation at https://developers.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling. Contact us for a consultation on your project and a budget estimate within 24 hours. Order integration today and get a free audit of your current AI pipeline.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.