尝试加载启用内存增长的模型时，Nvidia Xavier Jetson 中的张量流分段错误

如何解决尝试加载启用内存增长的模型时，Nvidia Xavier Jetson 中的张量流分段错误

我有一个非常具体的代码序列的分段错误，并且仅在 Xavier Jetson 上发生：

import os
import requests
import tensorflow as tf
  
# 1    
print('SET MEMORY GROWTH')
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0],True)  

# 2
print(f'REQUESTS GET')
requests.get('https://speed.hetzner.de/100MB.bin')

# 3
command = 'ls'
print(f'SYstem CALL ({command})')
os.system(command)

# 4 
print('MODEL LOAD') 
model = tf.keras.models.load_model('mnv2_xavier.h5')

如果我删除这些步骤之一，代码将毫无问题地运行。我不知道其他一些代码序列是否会导致同样的行为，但我很确定它们存在。

我想弄清楚这里出现分段错误的原因是什么，但到目前为止，我没有运气。

我认为这可能与 tensorflow 内存增长策略以及 Xavier Jetson 在 cpu 和 GPU 之间共享内存的事实有关。

我想知道是否有任何方法可以解决此问题或解决方法，以及是否有人对此行为有解释。

注意事项：

创建此模型的代码：

from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.models import Model
from tensorflow.keras import Input

x = Input((224,244,3))
y = MobileNetV2()(x)
model = Model(x,y)
model.save('mnv2_xavier.h5')

版本：

Jetpack 4.4
tensorflow 2.3.0
keras 2.4.0
python 3.6.9

输出：

2021-04-15 16:51:22.031610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
SET MEMORY GROWTH
2021-04-15 16:51:25.349940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-15 16:51:25.374098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.374309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.18GiB deviceMemoryBandwidth: 82.08GiB/s
2021-04-15 16:51:25.374437: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-15 16:51:25.377470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-15 16:51:25.379874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-15 16:51:25.380541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-15 16:51:25.383268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-15 16:51:25.385455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-15 16:51:25.385918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-15 16:51:25.386201: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386633: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
REQUESTS GET
SYstem CALL (ls)
code          logs          logs2
bashc.sh      main-log.log  tests
Desktop       Documents     mnv2_xavier.h5
Downloads     model.py      Music
Videos        Pictures      go  
Public        segfault.py 
MODEL LOAD
2021-04-15 16:51:29.542399: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine cpu frequency
2021-04-15 16:51:29.543521: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xcbba840 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-15 16:51:29.543595: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host,Default Version
Segmentation fault (core dumped)