[Tacotron2/Pytorch] when Inference on tritis, model size and model configuration size is different #474

Related to Model/Framework(s)
Pytorch/SpeechSynthesis/Tacotron2/notebooks/trtis
Describe the bug
A clear and concise description of what the bug is.
I followed all step of README.md in Pytorch/SpeechSynthesis/Tacotron2/notebooks/trtis
used pretrained data(Tacotron2: tacotron2_1032590_6000_amp, Waveglow: nvidia_waveglow256pyt_fp16)

just changing step in Setup WaveGlow TRT engine, I save engine name waveglow_fp16.engine not model.plan.
Also, In Dockerfile_tritis_client file change versions which is already installed in my server
RUN conda install -c conda-forge jupyterlab=1.1.4 ipywidgets=7.5.1 nodejs python-sounddevice librosa unidecode inflect

During the steps, there are some warnings occurs
step: export_waveglow_onnx
exports/export_waveglow_onnx.py:143: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! n_half = int(audio.size(1) / 2)

step: export_onnx2trt
[TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] ERROR: (Unnamed Layer* 170) [Slice]: slice size must be positive, size = [0,0,0] [TensorRT] WARNING: Calling isShapeTensor before the entire network is constructed may result in an inaccurate result. [TensorRT] WARNING: Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are: (# 0 (SHAPE sequences)) (# 1 (SHAPE sequences))
more warnings like these in export_onnx2trt

anyway It works
and I can run the TRTIS notebook client. and get URL. of jupyterLab
but when I run notebook.ipynb in jupyterLab , error occurs

---

InferenceServerException Traceback (most recent call last)
in
271
272 # default text
--> 273 text_area.value = "The forms of printed letters should be beautiful, and that their arrangement on the page should be reasonable and a help to the shapeliness of the letters themselves."

/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py in set(self, obj, value)
583 raise TraitError('The "%s" trait is read-only.' % self.name)
584 else:
--> 585 self.set(obj, value)
586
587 def _validate(self, obj, value):

/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py in set(self, obj, value)
572 # we explicitly compare silent to True just in case the equality
573 # comparison above returns something other than True/False
--> 574 obj._notify_trait(self.name, old_value, new_value)
575
576 def set(self, obj, value):

/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py in _notify_trait(self, name, old_value, new_value)
1137 new=new_value,
1138 owner=self,
-> 1139 type='change',
1140 ))
1141

/opt/conda/lib/python3.7/site-packages/ipywidgets/widgets/widget.py in notify_change(self, change)
604 # Send new state to front-end
605 self.send_state(key=name)
--> 606 super(Widget, self).notify_change(change)
607
608 def repr(self):

/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py in notify_change(self, change)
1174 c = getattr(self, c.name)
1175
-> 1176 c(change)
1177
1178 def _add_notifiers(self, handler, name, type):

in text_area_change(change)
235 display_heatmap(sequence)
236 # run tacotron2
--> 237 mel, mel_lengths, alignments = sequence_to_mel(sequence)
238 with plot_spectrogram:
239 display_spectrogram(mel, change['new'])

in sequence_to_mel(sequence)
133 batch_size = 1
134 # call tacotron2
--> 135 result = infer_ctx_tacotron2.run(input_dict, output_dict, batch_size)
136 # get results
137 mel = result['mel_outputs_postnet__0'][0] # take only the first instance in the output batch

/opt/conda/lib/python3.7/site-packages/tensorrtserver/api/init.py in run(self, inputs, outputs, batch_size, flags, corr_id)
1515
1516 # Run inference...
-> 1517 self._last_request_id = _raise_if_error(c_void_p(_crequest_infer_ctx_run(self._ctx)))
1518
1519 return self._get_results(outputs, batch_size)

/opt/conda/lib/python3.7/site-packages/tensorrtserver/api/init.py in _raise_if_error(err)
256 _crequest_error_del(err)
257 if not isok:
--> 258 raise ex
259 return ex.request_id()
260 return 0

InferenceServerException: [inference:0 11] output 'alignments__2' for model 'tacotron2' has shape [761] but model configuration specifies shape [-1,-1]

just hard coding, I changed shape of alignments__2 in trtis_repo/tacotron2/config.pbtxt file but it occurs another error
InferenceServerException: [inference:0 11] unexpected shape for output 'alignments__2', model configuration shape is [761], inference shape is [166,764]

I want to check there are some problem to make onnx or trt engine.
So, I follow quickstart of Pytorch/SpeechSynthesis/Tacotron2 and Pytorch/SpeechSynthesis/Tacotron2/trt examples using same onnx and trt engine.
but both of them run well and I get proper wav file.
So, I think there are no problem in onnx and trt engine.

To Reproduce
Steps to reproduce the behavior:

1. Install '...'
2. Set "..."
3. Launch '...'

Expected behavior
A clear and concise description of what you expected to happen.

Environment
Please provide at least:

• Container version (e.g. pytorch:19.05-py3): nvcr.io/nvidia/tensorrtserver:20.01-py3-clientsdk
• GPUs in the system: (e.g. 8x Tesla V100-SXM2-16GB): GeForce RTX 2080 Ti
• CUDA driver version (e.g. 418.67): CUDA Version 10.0.326