链接器培训经常失败

如何解决链接器培训经常失败

我正在Chainer中运行SSD300模型进行对象检测。 GPU是GeForce 2080 RTX Ti。在n个时期之后进行训练时，训练失败。它将引发以下错误：

Exception in main training loop: merge_sort: Failed to synchronize: an illegal memory access was encountered
Traceback (most recent call last):
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainer/training/trainer.py",line 316,in run
update()
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py",line 175,in update
self.update_core()
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py",line 187,in update_core
optimizer.update(loss_func,*in_arrays)
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainer/optimizer.py",line 800,in update
loss = lossfun(*args,**kwds)
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainer/link.py",line 294,in __call__
out = forward(*args,**kwargs)
  File "/home/malini/aim/top/process_runner1.py",line 291,in forward
mb_locs,mb_confs,gt_mb_locs,gt_mb_labels,self.k)
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainercv/links/model/ssd/multiBox_loss.py",line 91,in multiBox_loss
hard_negative = _hard_negative(conf_loss.array,positive,k)
  File "/home/malini/anaconda3/envs/chainer/lib/python3.7/site-packages/chainercv/links/model/ssd/multiBox_loss.py",line 19,in _hard_negative
rank = (x * (positive - 1)).argsort(axis=1).argsort(axis=1)
File "cupy/core/core.pyx",line 627,in cupy.core.core.ndarray.argsort
File "cupy/core/core.pyx",line 644,in cupy.core.core.ndarray.argsort
File "cupy/core/_routines_sorting.pyx",line 101,in cupy.core._routines_sorting._ndarray_argsort
File "cupy/cuda/thrust.pyx",line 135,in cupy.cuda.thrust.argsort
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx",line 193,in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx",line 82,in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEgal_ADDRESS: an illegal memory access was encountered

培训开始后遇到此错误。有时，培训会顺利完成，有时会失败。使用相同的数据集，batchsize和纪元时会发生这种情况。

我测试了使用减小的batchsize（4-32），发生了同样的事情。

在相同的情况下，它又怎么会失败而无法完成培训呢？

欢迎使用任何指针。