Skip to content

llama.cpp b12387, video input (ffmpeg and a host decoder, MediaMetadataRetriever on Android), allowUnauthenticatedTools() - #618

Merged
bernardladenthin merged 3 commits into
mainfrom
claude/wonderful-tesla-1fkezz
Nov 14, 2026
Merged

bernardladenthin merged 3 commits into
mainfrom
claude/wonderful-tesla-1fkezz

Conversation

@bernardladenthin

Copy link
Copy Markdown
Owner

Summary

Three commits:

  • Upgrade llama.cpp from b12316 to b12387 (a956a852). RPC protocol 10 (both ends need this llama.cpp again). --defrag-thold is gone upstream, so ModelParameters.setDefragThold is removed. New: allowPrivateMediaUrls(), the spec types draft-dflash2 and draft-lilicorr, CUDA arch 103-real. --cors-origins localhost means the server's own origin only; OriginGuard follows. server_queue::wait_until_no_sleep() now counts its caller as in flight, so the engine wakes the server through one helper and the user-reference guard gives the count back; without that a model with --sleep-idle-seconds never slept again after its first request (CApiSleep). Patches 0003 and 0015 refreshed (context only).
  • Video input over llama.cpp's own decoder, and allowUnauthenticatedTools(). ContentPart.videoFile / videoBytes / videoUrl, sent as video_url; LlamaModel.supportsVideo(); modalities.video in the Java server's /props; LangChain4j VideoContent mapped. The frames come from the ffmpeg / ffprobe programs llama.cpp runs as child processes; a machine without them refuses the request and goes on.
  • Video frames from the host. This project builds no decoder. Where ffmpeg cannot run, an existing library hands the frames in: jllama_model_load_video with a jllama_video_decoder (C API), callback.VideoDecoder / VideoStream and a LlamaModel constructor taking one (Java). On Android the llama-android AAR ships MediaRetrieverVideoDecoder over android.media.MediaMetadataRetriever; the AAR build therefore compiles against the platform's android.jar. The example app loads a vision model with it and offers video/* in its attach picker.

Test plan

  • Affected unit / integration tests pass locally: 793/793 C++ cases (CApiSleep, CApiVideo, CApiVideoDecoder, RpcComm on protocol 10, the CORS case); the Java video tests 11/11 against the generator's random vision model; LlamaArchitectureTest, SpotBugs, javadoc, check-natives.py, check-patches.py, the buildcheck unit tests, reuse lint
  • CI is green on this branch
  • Docs / CHANGELOG updated where applicable

Not run anywhere before this pull request, so CI is the first check:

  • the AAR build with the Android classes, the R8 consumer builds, and the Kotlin compile of the example app (the Android Gradle builds could not resolve their dependencies locally; the Java sources were compiled with javac against android.jar only)
  • OnDeviceVideoDecoderTest on the emulator
  • VideoInputIntegrationTest and VideoDecoderIntegrationTest with CI's SmolVLM pair (the model could not be downloaded locally)
  • every non-Linux platform

Open, recorded in TODO.md: no job runs a vision model on a device; the soundtrack (read_audio) is not passed through in Java; llama-ffm has no binding for the decoder; a video crosses as one byte[]; CUDA 13.5 with the new arch list is not measured on a GPU.

Related issues / PRs

None.

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes (if there are, I have notified the maintainer privately per SECURITY.md)

Generated by Claude Code

71 upstream commits, pinned by commit (upstream has not tagged the number
yet). Upstream versions this state 1.0.0 and pruned its public headers.

- The engine gives its wake back: server_queue::wait_until_no_sleep() now
  counts its caller as in flight (#32133), and a model loaded with
  --sleep-idle-seconds never slept again after its first call. One wake per
  thread (jllama_hold_server_awake), released by jllama_context_guard.
  CApiSleep is red without it.
- RPC protocol 10: the fake peer of test_rpc_comm.cpp follows.
- --cors-origins localhost serves the server's own origin only; OriginGuard
  and the tests of both servers follow upstream.
- ModelParameters.allowPrivateMediaUrls() (--media-allow-private-urls);
  setDefragThold removed with upstream's --defrag-thold; SpecType gains
  draft-dflash2 and draft-lilicorr.
- 103-real joins the CUDA architecture list of both CUDA jobs.
- Patches 0003 and 0015 refreshed (context only), 18 unchanged.
- The WebUI license generator reads a dependency resolved from GitHub's
  archive of a commit; build-webui needs no insteadOf rewrite any more.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
- ContentPart.videoFile / videoBytes / videoUrl put a video into a chat
  message (sent as video_url); LlamaModel.supportsVideo(). llama.cpp samples
  the frames and gives them to the vision projector; it decodes with the
  ffmpeg and ffprobe programs of the machine. This project adds no decoder.
- OpenAiCompatServer reports modalities.video in /props
  (OpenAiServerConfig.Builder.supportsVideo); the LangChain4j adapter maps
  VideoContent.
- Tests: CApiVideo (six cases over the random model and projector, clips
  ffmpeg writes at test time), VideoInputIntegrationTest (the vision model,
  a committed 3.5 KB clip; blocking, typed, streamed and HTTP paths, a
  machine without ffmpeg). The two Linux test jobs install ffmpeg and set
  JLLAMA_REQUIRE_FFMPEG, so the decoding cases cannot skip there.
- ModelParameters.allowUnauthenticatedTools()
  (--allow-unauthenticated-tools, llama.cpp b12387).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
…on Android

Where no ffmpeg program can be run, llama.cpp b12387 lets a host hand in the
frames (server_context::set_video_decoder). This passes it through; no codec
is implemented here.

- C API: jllama_video_decoder and jllama_model_load_video; the engine keeps
  the decoder for the life of the model and releases it exactly once, also
  when the load fails.
- Java: callback.VideoDecoder / VideoStream and
  LlamaModel(parameters, progress, decoder). The callbacks run on a thread
  of the library, attached to the JVM as "jllama-video" (measured), not on
  the caller's.
- llama-android: MediaRetrieverVideoDecoder over the platform's
  MediaMetadataRetriever, in the AAR's own package. The AAR build compiles it
  against a platform's android.jar (ANDROID_HOME or -PandroidJar).
- android-llmservice: a vision model is loaded with that decoder, and the
  attach picker offers videos (requirements R12.8).
- Tests: CApiVideoDecoder (C++) and VideoDecoderIntegrationTest (Java) with a
  stand-in decoder, no ffmpeg; OnDeviceVideoDecoderTest reads the committed
  clip through the platform decoder on the CI emulator. The post-release
  check and the AAR job know the AAR's package and require the class.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@bernardladenthin
bernardladenthin merged commit be50728 into main Nov 14, 2026
15 of 31 checks passed
@bernardladenthin
bernardladenthin deleted the claude/wonderful-tesla-1fkezz branch November 14, 2026 20:31
* @return {@code true} if this stream resamples to {@code framesPerSecond} itself
* @throws IOException when the stream cannot deliver frames; the request is refused
*/
default boolean start(int width, int height, float framesPerSecond) throws IOException {
* @return {@code true} if this stream resamples to {@code framesPerSecond} itself
* @throws IOException when the stream cannot deliver frames; the request is refused
*/
default boolean start(int width, int height, float framesPerSecond) throws IOException {
* @return {@code true} if this stream resamples to {@code framesPerSecond} itself
* @throws IOException when the stream cannot deliver frames; the request is refused
*/
default boolean start(int width, int height, float framesPerSecond) throws IOException {
@sonarqubecloud

Copy link
Copy Markdown

This branch was successfully deployed

1 active deployment
startgate — 4dc62f21 Deployed Nov 14, 2026 by bernardladenthin via Start gate (abort window) #1508
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants