Describe the bug
SpeechBrain LoRA adapter does not support convolutional layers, contrary to what is written here:
|
this adapter. Works with nn.Linear and nn.Conv |
Expected behaviour
The original LoRA paper does not address the case of convolutional layers, and this is quite uncommon in scientific publications. I suggest removing and nn.Conv from the line referenced above. Note that the HuggingFace peft library support it.
To Reproduce
The following model corresponds to the beginning of wav2vec2.0. For simplicity, I removed the LayerNorm and the GELU activation between conv1 and conv2.
import torch
import torch.nn as nn
from speechbrain.nnet.adapters import AdaptedModel, LoRA
class SimpleConv(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv1d(in_channels=1, out_channels=512, kernel_size=10, stride=5)
self.conv2 = nn.Conv1d(in_channels=512, out_channels=512, kernel_size=3, stride=2)
def forward(self, x):
return self.conv2(self.conv1(x))
model = SimpleConv()
adapted_model = AdaptedModel(
model_to_adapt=model,
adapter_class=LoRA,
adapter_kwargs={"rank": 16},
target_layers=["*conv2"]
)
x = torch.randn(1, 1, 5000)
out = adapted_model(x)Environment Details
No response
Relevant Log Output
Traceback (most recent call last):
File "lora_bug.py", line 24, in <module>
out = adapted_model(x)
^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "speechbrain/speechbrain/nnet/adapters.py", line 147, in forward
return self.adapted_model(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "lora_bug.py", line 11, in forward
return self.conv2(self.conv1(x))
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "speechbrain/speechbrain/nnet/adapters.py", line 395, in forward
x_lora = self.adapter_up_proj(self.adapter_down_proj(x)) * self.scaling
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File ".venv/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 134, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x999 and 512x16)Additional Context
No response
Describe the bug
SpeechBrain LoRA adapter does not support convolutional layers, contrary to what is written here:
speechbrain/speechbrain/nnet/adapters.py
Line 335 in 8a89eba
Expected behaviour
The original LoRA paper does not address the case of convolutional layers, and this is quite uncommon in scientific publications. I suggest removing
and nn.Convfrom the line referenced above. Note that the HuggingFacepeftlibrary support it.To Reproduce
The following model corresponds to the beginning of wav2vec2.0. For simplicity, I removed the LayerNorm and the GELU activation between conv1 and conv2.
Environment Details
No response
Relevant Log Output
Additional Context
No response