Keras residual connection on only part of input

Question

I already made a model without residual connection which compile and fit without any errors [using Keras Sequential API]

I wish to test a modified version just adding a residual connection like in SPEECH ENHANCEMENT BASED ON DEEP NEURAL NETWORKS WITH SKIP CONNECTIONS. So, I need to use Functional API instead.

My problem is extract a piece in the middle of inputs. I tried that.

INPUT_SIZE = N*OUTPUT_SIZE # N must be odd
HIDDEN_SIZE = N*OUTPUT_SIZE # N must be odd

modelInputs = Input(shape=(INPUT_SIZE,))
x = Dense(HIDDEN_SIZE, activation='relu', kernel_initializer=INPUT_KERNEL_INITIALIZER)(modelInputs)
for _ in np.arange(1,N_HIDDEN):
    x = Dense(HIDDEN_SIZE, activation='relu', kernel_initializer=INPUT_KERNEL_INITIALIZER)(x)
Y = Dense(OUTPUT_SIZE, activation='relu', kernel_initializer=INPUT_KERNEL_INITIALIZER)(x)

# --------------------------------------------------------
# Here, 4 options I tried to get "modelInputs_selected"
# --------------------------------------------------------
# Try 1
modelInputs_selected = Lambda(lambda x: x[int(N/2)*OUTPUT_SIZE:(int(N/2)+1)*OUTPUT_SIZE])(modelInputs)
# Try 2 [Try 1 with 'output_shape' filled]
modelInputs_selected = Lambda(lambda x: x[int(N/2)*OUTPUT_SIZE:(int(N/2)+1)*OUTPUT_SIZE, :], output_shape=(OUTPUT_SIZE,))(modelInputs)

# Try 3
modelInputs_selected = K.transpose(K.gather(K.transpose(modelInputs), K.arange(int(N/2)*OUTPUT_SIZE, (int(N/2)+1)*OUTPUT_SIZE)))
# Try 4 [Try 3 unwrapped]
toto1 = K.transpose(modelInputs)
toto2 = K.gather(toto1, K.arange(int(N/2)*OUTPUT_SIZE, (int(N/2)+1)*OUTPUT_SIZE))
modelInputs_selected = K.transpose(toto2)
# --------------------------------------------------------
# End of option tried
# --------------------------------------------------------

predictions = add([modelInputs_selected, Y])

model = Model(inputs=modelInputs, outputs=predictions)

Results are:

Try 1 & Try 2:
- Error = Incoherent shape during add()
Try 3 & Try 4:
- Good shapes for add()
- Error with Model(...)
  - I gone into Model() step by step. We go backward starting with the last layer
  - Output add() OK
  - Previous one K.transpose(): Error AttributeError: 'Tensor' object has no attribute '_keras_history' in "build_map_of_graph"

The model construction failed because I use a function from the backend (TensorFlow, for me)?

Anyone can help, please?

Maybe if I use a multiply()?

modelInputs is (m, N*OUTPUT_SIZE) and modelInputs_selected is (m,OUTPUT_SIZE)
With the good matrix A (N*OUTPUT_SIZE, OUTPUT_SIZE): modelInputs_selected = multiply(modelInputs, A)

Daniel Möller · Accepted Answer · 2018-03-09 16:27:28Z

0

Error in try 1

You're ignoring the samples dimension (the first dimension).
The tensor x should come in with shape (samples_or_batch_size, INPUT_SIZE).

Solution:
- So you need lambda x: x[:, int(N/2)*OUTPUT_SIZE:(int(N/2)+1)*OUTPUT_SIZE]
But honestly, why not one of these?
- lambda x: x[:,:OUTPUT_SIZE]
- lambda x: x[:,-OUTPUT_SIZE:]

Is it a problem taking from the beginning or from the end? This even frees you from using an N, the only condition will be that INPUT_SIZE >= OUTPUT_SIZE.

Error in try 2

You're not ignoring the samples, but you're inverting the dimensions.
It should be x[:,expression] instead of x[expression,:].

You only need to declare the output shape if you're not using Tensorflow (Tensorflow can do it automatically for you)

Errors in 3 and 4

You cannot use any function outside of a layer.
There is no problem if you use either Keras backend or Tensorflow functions, but they must be inside layers, such as in try 1 and 2.

The no _keras_history is a typical error for using functions outside of layers.

edited Mar 9, 2018 at 16:27

answered Mar 9, 2018 at 16:21

Daniel Möller

86.8k24 gold badges202 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

AlexDtd Over a year ago

Thanks. Why I want to use the middle one: My inputs are N FFTs ("half' in the past, current one and "half" in the future) and I wish to use the "current time" FFT. This organisation is more for logical presentation of datas because, of course", I should put the current FFT in first position ...

Collectives™ on Stack Overflow

Keras residual connection on only part of input

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related