Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: SKaiNET-developers/SKaiNET-transformers
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: develop
Choose a base ref
...
head repository: SKaiNET-developers/SKaiNET-transformers
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: feature/qwen35
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 5 commits
  • 20 files changed
  • 2 contributors

Commits on Apr 21, 2026

  1. Add initial implementaion of DeltaNet, not working yet

    @michalharakal
    michalharakal committed Apr 21, 2026
    Configuration menu
    Copy the full SHA
    5b3dc6d View commit details
    Browse the repository at this point in the history

  2. fix: correct DeltaNet forward pass and full attention gate in Qwen35R… 

    …untime

The DeltaNet implementation produced garbled output due to 9 bugs identified
by cross-referencing with llama.cpp and the NVlabs GatedDeltaNet reference:

- Fix QKV split from 2-way [Q:4096,K:4096] to 3-way [Q:2048,K:2048,V:4096]
- Fix conv1d weight indexing to channel-major layout (channel*d_conv+tap)
- Implement delta rule recurrence with prediction error (was missing V entirely)
- Fix decay formula: softplus(alpha+dt_bias)*ssm_a instead of sigmoid
- Add L2 normalization on Q and K per head after conv+SiLU
- Add Q/K head repetition from 16 to 32 to match V heads
- Fix output path order: RMSNorm → SiLU gate → project (was project → gate)
- Add sigmoid gate from joint Q+Gate projection in full attention layers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
    @michalharakal @claude
    michalharakal and claude committed Apr 21, 2026
    Configuration menu
    Copy the full SHA
    4af7e99 View commit details
    Browse the repository at this point in the history

  3. fix: split Q+Gate before norm/RoPE in full attention, add Qwen35Runti… 

    …me tests

The full attention path applied RMSNorm and RoPE to the joint Q+Gate
projection (16 heads x 512 dims) instead of splitting first and norming
only Q (16 heads x 256 dims). This corrupted both Q values and gate
values, producing garbage from every full attention layer.

Also refactored applyPerHeadRMSNorm to work on raw FloatArrays to
avoid unnecessary tensor wrap/unwrap in the split path.

Added Qwen35RuntimeTest with 11 unit tests using a tiny synthetic model
(dim=16, 4 layers, vocab=8) that exercises all key operations without
loading a GGUF file: 3-way QKV split, conv1d weight indexing, delta rule
recurrence, L2 normalization, head repetition, Q+Gate split, sigmoid
gating, and state boundedness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
    @michalharakal @claude
    michalharakal and claude committed Apr 21, 2026
    Configuration menu
    Copy the full SHA
    5bdab52 View commit details
    Browse the repository at this point in the history

  4. chore: add init-time dimensional validation and diagnostic prints 

    Add require() checks in Qwen35Runtime init to catch dimensional
mismatches early (QKV split, V-head count, head ratio). Also print
GGUF tensor shapes and matmul convention for debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
    @michalharakal @claude
    michalharakal and claude committed Apr 21, 2026
    Configuration menu
    Copy the full SHA
    33efe46 View commit details
    Browse the repository at this point in the history

  5. feat: add tool calling smoke tests and fix GGUFTokenizer byte-level BPE 

    Extend smoke tests with a tool calling phase for models that support it
(Qwen3, Qwen3.5). Add single-shot mode to ToolCallingDemo so it can run
non-interactively via --demo with a positional prompt argument.

Fix GGUFTokenizer to correctly encode chat template prompts for GPT-2/Qwen
models by adding: special token splitting (<|im_start|>, <|im_end|>),
byte-level BPE with merge ranks from tokenizer.ggml.merges, and GPT-2
pretokenization regex to prevent cross-word merges.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
    @michalharakal @claude
    michalharakal and claude committed Apr 21, 2026
    Configuration menu
    Copy the full SHA
    5637da3 View commit details
    Browse the repository at this point in the history
Loading