The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.
text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.
For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.
All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.
Got it, I'll test more deployment params when I benchmark models for vision tasks.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
Have you tried any vision tasks with this model? We've been serving these on our gateway [1], and the quants are quite terrible for vision. Curious to hear your experience.
VLM Run (https://vlm.run) | 1x Founding Infrastructure Engineer
We’re building the inference platform for visual intelligence. We’re a deeply technical team of veteran AI / computer-vision engineers (20+ years combined, MIT/CMU PhDs) who’ve shipped production ML infrastructure across autonomous driving and LLMs.
We just launched the VLM Run Gateway (https://vlm.run/gateway), a unified API that serves open-weight VLMs, embodied VLAs, ViTs, served across a fleet of GPUs and clouds. That's exactly the infrastructure problem this role will own.
If you’re interested, email us at hiring@vlm.run with your GitHub profile and link to projects you’ve recently built - especially with docker, k8s, GPUs.
I think we all ought to look at the ZDR fine-print here.
I get that in principle that there's no retention, but these are powerful models that can comprehend, paraphrase and summarize your logs for the sake of "product" improvement. Who knows what's collected here.
It’s unclear to me what their desired outcome for a blog post like this. If you’ve ever worked in a robotics setting, 80% implies that 20% of your autonomous actions are incorrect. Imagine if this were the case for autonomous driving where your car misbehaves 1 in every 5 actions it takes.
Posts like this just reminds me of the end to end demos AV companies built in the early days using a single camera - only to realize that it’s harder than it looks years later into development.
The ICP question was more around the model itself. Are they looking to license it to robotics companies? Do they imagine that devs at robotics companies would be willing to deploy these models as a black box?
Really, does this work now? What about NotebookLM? I was using it a lot until i realised it was only analysing the transcripts and not the video because i was mostly using it for technical ones with important charts.
NotebookLM still uses the transcript method I think. But Gemini is wonderful. I have been using it to analyze the youtube videos of wrestling matches (trying to build a fan website for WXM, the best pro wrestling promotion to come out of India in a while). It does move by move analysis, audience reaction based match flow tracking, isolates interesting parts of the video (big moves, botches, story beats etc). I have run some experiments to get video editing plans out of it. I think I can combine it with something like remotion skill to make highlight videos.
Edit: BTW, you can analyze about 8 hours a day on free tier.
NotebookLM was transcript-only when I last checked. Gemini proper does ingest video natively (samples ~1fps server-side). This tool is for everything that can't — Claude, ChatGPT web, local models — it turns the video into frames + transcript on your machine so any of them can read it.
It can tell you what’s on the screen at given point in time. My pipeline is mostly around simple questions like “does this video contain cars?” Not sure if it can spot charts on screen.
Do you mean that Gemini is most token-efficent at watching videos? Is that the case for e.g. just giving it a video in the browser? I admit, I dont give LLMs videos as I just assume it'll burn too many tokens.
Yes, Gemini is very token efficient at video. It also has "lower resolution" options which can make it even cheaper if. With Gemini 3.1 flash lite an hour of video works out to $0.24 at the API rates.
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.
reply