391 questions
Advice
0
votes
2
replies
48
views
Fixing a UNET in pytorch that doesn't work in eval mode due to BatchNorm2d layers
I have a UNET model trained in pytorch (by someone else) that produces quite different results in eval mode to train mode (train mode results look good, eval mode they are rubbish). A bit of googling ...
- 11
2
votes
0
answers
78
views
BatchNormalization mismatch between Pytorch and Keras
I have built a U-Net like architecture called DF-Net for image manipulation localization. I implemented the model in Pytroch but found later the model's saved metadata and so on in Keras format. I ...
- 21
1
vote
2
answers
159
views
Changing BatchNormalization momentum while training in Keras 3
I'm buidling a custom model using Keras (keras 3.3.3 with python 3.9.19) and I would like to increase the momentum of my BatchNormalization layers during training.
Ideally, I'd like to use a custom ...
- 187
1
vote
0
answers
36
views
Shape of Data2Vec output dimensions
I'm using Data2Vec from the Huggingface hub to feature extract on three modalities of a dataset. After processing I have tensors the shape of [1,768] for text, [1,499,768] for audio, and [1,197,768] ...
- 11
0
votes
0
answers
160
views
Batch normalization in neural network for regression task
I have this neural network that I want to fit for regression. When I don't use batch normalization between layers it performs good, but when I add batch normalization it performs really bad.
For ...
- 127
0
votes
0
answers
50
views
How to rectify this mismatch error in the case of Neural Surrogates BatchNormalization
x_train =
100-element Vector{Tuple{Float64, Float64}}:
(0.1875, 3.1875)
(4.1875, 7.1875)
(6.1875, 1.1875)
(2.1875, 5.1875)
(3.1875, 0.1875)
(7.1875, 4.1875)
(5.1875, 2.1875)
(1.1875, 6.1875)
...
0
votes
1
answer
982
views
Batchnorms force set to training mode on torch.onnx.export when running stats are None
As described in this git issue (very complete description), I'm trying to load an .onnx model in openvino backend. However, the BatchNorm layers are considered in training mode when setting ...
- 37
1
vote
1
answer
373
views
Pytorch batchnorm2d: "RuntimeError: running_mean should contain 1 elements not 64"
None of the similar questions worked. So please do not flag as dublicate
Pytorch BatchNorm2d expects an input in the format N C H W
where
N = Batchsize
C = Channels
H = Height
W = Width
as they ...
- 184
1
vote
0
answers
215
views
SpectralNorm gets cancelled when followed by BatchNorm?
I am implementing a version of FastGAN, and it seems like (see the official repo) they use a Spectral norm directly followed by a Batch norm. Doesn't the spectral norm get fully cancelled by the ...
- 56
2
votes
0
answers
454
views
Why does LayerNorm use a biased standard deviation estimator?
The LayerNorm computation in the original paper Layer Normalization uses a biased estimator of standard deviation (see equation 3 below). Why does it use a biased estimator instead of an unbiased ...
- 1,826
0
votes
1
answer
115
views
How can I reinitialize the Batchnorm Layer?
I have a CNN with batchnorm layers. I am trying to train the CNN for a few epochs, and then I want to reset the batchnorm weights (moving_mean and moving_variance) while preserving the learned CNN ...
- 136
0
votes
0
answers
292
views
Can't fold BatchNorm with Conv2D in Keras QAT basic example
I'm currently trying to use Keras' Quantization Aware Training, specifically because I need to do 8bit inference on a low-precision device. For this reason, I need to fold the batch norm onto the ...
- 1
1
vote
1
answer
244
views
Initializing two neural networks from the same class: first initialization influences the second
I'm a begginer with PyTorch and I'm attempting to implement student-teacher architecture by initializing two networks with different hidden sizes from the same class. It seems the first network ...
- 13
0
votes
1
answer
162
views
BatchNorm1d channel size cannot match
I get an error when trying to implement batchnorm1d in the model. The error saying RuntimeError: running_mean should contain 32 elements not 256, so I print out the x size after pass through fc1 which ...
- 119
0
votes
1
answer
609
views
pytorch instance normalization, batch normalization (training) and onnx MVN and openvino MVN
What is the difference between Instance normalization and MVN and Batch normalization (training mode and batch_size=1) in onnx?
My pytorch model is currently using instance normalization but my ...
- 79
0
votes
1
answer
736
views
Minimum Batch Size vs Batch Normalisation
I have been using VGG16 with transfer learning to for training a CNN via PyTorch. The maximum batch size that I can use is 16 as I am constrained by VRAM (GTX 1070).
I have noticed that when training ...
0
votes
1
answer
78
views
Is mp.Queue that limited in memory?
So, I've impemented my synchronous batch normalization layer. I've tried to test it by running on differenet set of parameters by this way: each mp.Process runs a function, which makes forward and ...
0
votes
1
answer
2k
views
How to use PyTorch nn.BatchNorm1d to get equal normalization across features?
i would like to ask a question regarding the nn.BatchNorm1d in PyTorch.
I have one main tensor, which has shape [B, 3, N]. Then, i have two additional tensors which have shape [B, 3, V1] and [B, 3, V2]...
- 39
0
votes
1
answer
163
views
Reshape data to be usable for training GCN in PyTorch
I am trying to build Graph Convolutional Network. I converted my dataframe to PyTorch
required format using below code.
class S_Dataset(Dataset):
def __init__(self, df, transform=None):
self.df = ...
- 51
0
votes
1
answer
80
views
Can we use normCounts(RUVr) output to calculate TPM?
I have used RUVr() from RUVSeq R packages, to correct batch in my data, now I would like to use the corrected table to calculate TPM.
Can I use normCounts() to extract batch corrected data for TPM?
- 775
3
votes
0
answers
207
views
Batch Normalization Vs. Instance Normalization Vs. Adaptive Instance Normalization
What is the difference between Batch Normalization, Instance Normalization, Adaptive Instance Normalization layers in a CNN?Which one should be used in generative models for image stylization?
I am ...
- 31
1
vote
1
answer
650
views
Can we avoid specify "num_features" in batch norm of PyTorch, just like Tensorflow?
Here is batch norm in TF:
model = BatchNormalization(momentum=0.15, axis=-1)(model)
And here is batch norm in Torch:
torch.nn.BatchNorm1d(num_features, eps=1e-05, momentum=0.1, affine=True, ...
- 165
1
vote
0
answers
97
views
Why does the accuracy fluctuate widely after using batch normalization
I'm training a model which includes batch normalization layer, but i noticed that the accuracy can fluctuate widely (from 55% to 31% in just one epoch), both train accuracy and test accuracy, so i ...
- 43
0
votes
1
answer
609
views
Pytorch Expected more than 1 value per channel when training when using BatchNorm
I've written this code:
import numpy as np
import torch
from torch.utils.data import TensorDataset, dataloader
inputDim = 10
n = 1000
X = np.random.rand(n,inputDim)
y = np.random.rand(0,2,n)
...
- 1,258
1
vote
1
answer
269
views
Why doesn't a batch-normalized layer sum to 1?
I've been looking deeper into how batch norm works in PyTorch and have noticed that for the below code:
torch.manual_seed(0)
# With Learnable Parameters
m = nn.BatchNorm2d(1)
# Without Learnable ...
- 603
1
vote
1
answer
505
views
How to implement batch normalization merging in python?
I have defined the model as in the code below, and I used batch normalization merging to make 3 layers into 1 linear layer.
The first layer of the model is a linear layer and there is no bias.
The ...
- 870
0
votes
1
answer
80
views
Error in pnbinom(counts_sub[a, b] - 1, mu = old_mu[a, b], size = 1/old_phi[a])
I'm trying to perform batch correction on raw RNA-Seq counts. However, when I try to run ComBat, this is what happens:
library("sva")
batch <- c(rep(1, 9),rep(2, 19))
adjusted <- ...
0
votes
2
answers
1k
views
model.predict() - TensorFlow Keras gives same output for all images when the dataset size increases?
I have been trying to use a pre-trained model(XceptionNet) to get a feature vector corresponding to each input image for a classification task. But am stuck as the model.predict() gives unreliable and ...
2
votes
0
answers
325
views
Model not training when using batch normalization with keras functional API
I'm going through some tutorials using the Keras functional API in Tensorflow 2, and I'm having some trouble including BatchNormalization layers when using the functional API.
Using roughly the same ...
1
vote
1
answer
970
views
Instance Norm: ValueError: Expected more than 1 spatial element when training, got input size torch.Size([128, 512, 1, 1])
I have a ResNet-18 working well. Now, I want to use InstanceNorm as normalization layer instead of BatchNorm, so I changed all the batchnorm layers in this way:
resnet18.bn1 = nn.InstanceNorm2d(64, ...
- 478
0
votes
0
answers
108
views
Building model matrix to correct for batch effect with biological and technical replicates
I recently conducted some MASS SPEC for my samples. Each sample was run thrice through the machine. However, there was a large space of time between the first run and the consequent second and third ...
0
votes
1
answer
2k
views
RuntimeError: Expected number of channels in input to be divisible by num_groups, but got input of shape [64, 16, 32, 32] and num_groups=32
I have EfficientNet working fine on my dataset. Now, I changed all the batch norm layers into group norm layers. I have already done this process with other networks like vgg16 and resnet18 and all ...
- 478
1
vote
0
answers
390
views
Tensorflow Batch Normalization results dependent on number of channels in input
I'm trying to port weights for batch normalization from tensorflow to pytorch and I encountered a strange issue in tensorflow. The below code excerpt computes batchnorm with 10 channels with input of ...
1
vote
1
answer
386
views
how to convert tf.contrib.layers.batch_norm to tf2.0
I am converting the following code
import tensorflow.compat.v1 as tf
def conv_layer(input_tensor,name,kernel_size,output_channels,initializer=tf.keras.initializers.VarianceScaling,stride=1,bn=False,...
- 11
2
votes
1
answer
1k
views
Pytorch: Disable only nn.Dropout() without using model.eval()
nn.Dropout() can be disabled by using model.eval().However by using .eval(), nn.BatchNorm1d() are also disabled. Because the distributions between train and test sets are different, I'd like to ...
- 21
4
votes
1
answer
8k
views
What do BatchNorm2d's running_mean / running_var mean in PyTorch?
I'd like to know what exactly the running_mean and running_var that I can call from nn.BatchNorm2d.
Example code is here where bn means nn.BatchNorm2d.
vector = torch.cat([
torch.mean(self.conv3....
- 101
0
votes
1
answer
672
views
Batch Normalization in 4D data over features instead of channels (PyTorch)
I am trying to implement batch normalization in my CNN via nn.BatchNorm2d. My data is of size (N, C, H, W) where N is the batch size, C is the number of channels, and HxW is the image size. Now in 1d ...
- 473
0
votes
3
answers
568
views
Why is keras and BatchNormalization underlined?
I am trying to run a CNN python code, but at the top of the code, the following line has keras and BatchNormalization underline in red.
from keras.layers.normalization import BatchNormalization
I don'...
- 105
0
votes
1
answer
2k
views
cannot import name 'BatchNormalization' from 'keras.layers.normalization'
I'm learning ObjectDetection from this website
I have installed ImageAI,Tensorflow and Keras.
Then when I run this in python
from imageai.Detection import ObjectDetection
I got
Traceback (most ...
2
votes
0
answers
417
views
'RaggedTensor' object has no attribute 'set_shape' when passing RaggedTensor to BatchNormalization layer in Tensorflow 2.8
I am trying to pass a 3D RaggedTensor of shape (batch_size=None, 1, 136) to a batch normalization layer, but am receiving the error in the title. I assume because RaggedTensors can very in dimensions, ...
- 684
1
vote
0
answers
98
views
BatchNormalization Bijector Wrong Result with "prob" method
I am trying to implement a normalizing flow according to the RealNVP model for density estimation.
First, I am trying to make it work on the "moons" toy dataset.
The model produces the ...
2
votes
1
answer
2k
views
Is it a good idea to train BatchNorm layer with running stats?
As far as I know, BatchNorm will use batch stats in train mode, but use running stats (running_mean/running_var) in eval mode. How about just always use running stats in both train and eval mode?
In ...
- 101
1
vote
1
answer
841
views
Not able to switch off batch norm layers for faster-rcnn (PyTorch)
I'm trying to switch off batch norm layers in a faster-rcnn model for evaluation mode.
I'm doing a sanity check atm:
@torch.no_grad()
def evaluate_loss(model, data_loader, device):
val_loss = 0
...
- 546
2
votes
1
answer
921
views
Tensorflow keras BatchNormalization for higher than 4-dimension Tensor (video input)
I'm trying to implement S3D[https://arxiv.org/pdf/1712.04851.pdf] for video classification and I encountered a problem with BatchNormalization.
Since the implementation that I'm dealing with is video ...
- 41
1
vote
1
answer
280
views
Layer normalization Weight re-scaling and re-centering
In the paper on Layer normalization (https://arxiv.org/pdf/1607.06450.pdf) they say that
layer normalization is invariant to scaling of the entire weight
matrix and invariant to a shift to all of the ...
- 43
2
votes
1
answer
502
views
Does batch normalization replace 'layers.experimental.preprocessing.Rescaling' in CNN models?
Does batch normalization replace 'layers.experimental.preprocessing.Rescaling' in CNN models?
Or we should first normalize the data and then use BN in the CNN model.
- 21
1
vote
1
answer
953
views
batch size without batch normalization
I'm working on image super-resolution tasks with EDSR as a baseline model. Following EDSR, I'm not using any batch-norm layers in my model. I suddenly came up with a stupid question about batch-sizes.
...
- 37
0
votes
2
answers
2k
views
Using BatchNorm1d layer with Embedding and Linear layers for NLP text-classification problem throws RuntimeError
I am trying to create a neural network and train my own Embeddings. The network has the following structure (PyTorch):
import torch.nn as nn
class MultiClassClassifer(nn.Module):
#define all the ...
- 1,589
1
vote
1
answer
2k
views
Pytorch running_mean, running_var and num_batches_tracked are updated during training, but I want to fix them
In pytorch, I want to use a pretrained model and train my model to add a delta to the model result, that is:
╭----- (pretrained model) ------ result ---╮
input------------- (my model) --------...
- 11
0
votes
2
answers
1k
views
Changing BatchNormalization momentum while training in Tensorflow 2
I want batch normalization running statistics (mean and variance) to converge in the end of training, which requires to increase batch norm momentum from some initial value to 1.0. I managed to change ...
- 629