Repository navigation
llama.cpp b12387, video input (ffmpeg and a host decoder, MediaMetadataRetriever on Android), allowUnauthenticatedTools() - #618
Merged
Conversation
71 upstream commits, pinned by commit (upstream has not tagged the number yet). Upstream versions this state 1.0.0 and pruned its public headers. - The engine gives its wake back: server_queue::wait_until_no_sleep() now counts its caller as in flight (#32133), and a model loaded with --sleep-idle-seconds never slept again after its first call. One wake per thread (jllama_hold_server_awake), released by jllama_context_guard. CApiSleep is red without it. - RPC protocol 10: the fake peer of test_rpc_comm.cpp follows. - --cors-origins localhost serves the server's own origin only; OriginGuard and the tests of both servers follow upstream. - ModelParameters.allowPrivateMediaUrls() (--media-allow-private-urls); setDefragThold removed with upstream's --defrag-thold; SpecType gains draft-dflash2 and draft-lilicorr. - 103-real joins the CUDA architecture list of both CUDA jobs. - Patches 0003 and 0015 refreshed (context only), 18 unchanged. - The WebUI license generator reads a dependency resolved from GitHub's archive of a commit; build-webui needs no insteadOf rewrite any more. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
- ContentPart.videoFile / videoBytes / videoUrl put a video into a chat message (sent as video_url); LlamaModel.supportsVideo(). llama.cpp samples the frames and gives them to the vision projector; it decodes with the ffmpeg and ffprobe programs of the machine. This project adds no decoder. - OpenAiCompatServer reports modalities.video in /props (OpenAiServerConfig.Builder.supportsVideo); the LangChain4j adapter maps VideoContent. - Tests: CApiVideo (six cases over the random model and projector, clips ffmpeg writes at test time), VideoInputIntegrationTest (the vision model, a committed 3.5 KB clip; blocking, typed, streamed and HTTP paths, a machine without ffmpeg). The two Linux test jobs install ffmpeg and set JLLAMA_REQUIRE_FFMPEG, so the decoding cases cannot skip there. - ModelParameters.allowUnauthenticatedTools() (--allow-unauthenticated-tools, llama.cpp b12387). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
…on Android Where no ffmpeg program can be run, llama.cpp b12387 lets a host hand in the frames (server_context::set_video_decoder). This passes it through; no codec is implemented here. - C API: jllama_video_decoder and jllama_model_load_video; the engine keeps the decoder for the life of the model and releases it exactly once, also when the load fails. - Java: callback.VideoDecoder / VideoStream and LlamaModel(parameters, progress, decoder). The callbacks run on a thread of the library, attached to the JVM as "jllama-video" (measured), not on the caller's. - llama-android: MediaRetrieverVideoDecoder over the platform's MediaMetadataRetriever, in the AAR's own package. The AAR build compiles it against a platform's android.jar (ANDROID_HOME or -PandroidJar). - android-llmservice: a vision model is loaded with that decoder, and the attach picker offers videos (requirements R12.8). - Tests: CApiVideoDecoder (C++) and VideoDecoderIntegrationTest (Java) with a stand-in decoder, no ffmpeg; OnDeviceVideoDecoderTest reads the committed clip through the platform decoder on the CI emulator. The post-release check and the AAR job know the AAR's package and require the class. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8cAMZXymmPba3YZr2YCWp
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
| * @return {@code true} if this stream resamples to {@code framesPerSecond} itself | ||
| * @throws IOException when the stream cannot deliver frames; the request is refused | ||
| */ | ||
| default boolean start(int width, int height, float framesPerSecond) throws IOException { |
| * @return {@code true} if this stream resamples to {@code framesPerSecond} itself | ||
| * @throws IOException when the stream cannot deliver frames; the request is refused | ||
| */ | ||
| default boolean start(int width, int height, float framesPerSecond) throws IOException { |
| * @return {@code true} if this stream resamples to {@code framesPerSecond} itself | ||
| * @throws IOException when the stream cannot deliver frames; the request is refused | ||
| */ | ||
| default boolean start(int width, int height, float framesPerSecond) throws IOException { |
|
4 of 6 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Three commits:
a956a852). RPC protocol 10 (both ends need this llama.cpp again).--defrag-tholdis gone upstream, soModelParameters.setDefragTholdis removed. New:allowPrivateMediaUrls(), the spec typesdraft-dflash2anddraft-lilicorr, CUDA arch103-real.--cors-origins localhostmeans the server's own origin only;OriginGuardfollows.server_queue::wait_until_no_sleep()now counts its caller as in flight, so the engine wakes the server through one helper and the user-reference guard gives the count back; without that a model with--sleep-idle-secondsnever slept again after its first request (CApiSleep). Patches0003and0015refreshed (context only).allowUnauthenticatedTools().ContentPart.videoFile/videoBytes/videoUrl, sent asvideo_url;LlamaModel.supportsVideo();modalities.videoin the Java server's/props; LangChain4jVideoContentmapped. The frames come from theffmpeg/ffprobeprograms llama.cpp runs as child processes; a machine without them refuses the request and goes on.jllama_model_load_videowith ajllama_video_decoder(C API),callback.VideoDecoder/VideoStreamand aLlamaModelconstructor taking one (Java). On Android thellama-androidAAR shipsMediaRetrieverVideoDecoderoverandroid.media.MediaMetadataRetriever; the AAR build therefore compiles against the platform'sandroid.jar. The example app loads a vision model with it and offersvideo/*in its attach picker.Test plan
CApiSleep,CApiVideo,CApiVideoDecoder,RpcCommon protocol 10, the CORS case); the Java video tests 11/11 against the generator's random vision model;LlamaArchitectureTest, SpotBugs, javadoc,check-natives.py,check-patches.py, the buildcheck unit tests,reuse lintNot run anywhere before this pull request, so CI is the first check:
javacagainstandroid.jaronly)OnDeviceVideoDecoderTeston the emulatorVideoInputIntegrationTestandVideoDecoderIntegrationTestwith CI's SmolVLM pair (the model could not be downloaded locally)Open, recorded in
TODO.md: no job runs a vision model on a device; the soundtrack (read_audio) is not passed through in Java;llama-ffmhas no binding for the decoder; a video crosses as onebyte[]; CUDA 13.5 with the new arch list is not measured on a GPU.Related issues / PRs
None.
Checklist
CONTRIBUTING.mdandCODE_OF_CONDUCT.mdSECURITY.md)Generated by Claude Code