RCNN模型内存耗尽

如何解决RCNN模型内存耗尽

我正在尝试在自己的数据集上运行R-CNN模型，但是遇到以下错误：

2020-10-21 20:44:19.018471: W tensorflow/core/common_runtime/bfc_allocator.cc:424] *************************************************xx****x_**********************__************_******
2020-10-21 20:44:19.019036: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES Failed at bias_op.cc:507 : Resource exhausted: OOM when allocating tensor with shape[512,65536] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

我可以在控制台中找到我的设备

2020-10-21 20:39:16.560354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62

数据集由一个包含8000张图像的文件夹组成。图片大小介于512x512到1024x1024之间。 每个GPU的图像设置为1，STEPS_PER_EPOCH设置为3000。例如，如果我将STEPS_PER_EPOCH设置为80，则不会出错。我无法理解此参数是指一次处理的图像还是每个时期要处理的图像数。 该电脑具有16 GB的内存，其中4GB专用，8GB共享。

ERROR MESSAGE:


    ResourceExhaustedError: 2 root error(s) found.
      (0) Resource exhausted: OOM when allocating tensor with shape[200,256,14,14] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node mrcnn_mask_conv4_3/convolution}}]]
    Hint: If you want to see a list of allocated tensors when OOM happens,add report_tensor_allocations_upon_oom to Runoptions for current allocation info.

     [[mul_206/_18183]]
Hint: If you want to see a list of allocated tensors when OOM happens,add report_tensor_allocations_upon_oom to Runoptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[200,14] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node mrcnn_mask_conv4_3/convolution}}]]
Hint: If you want to see a list of allocated tensors when OOM happens,add report_tensor_allocations_upon_oom to Runoptions for current allocation info.

0 successful operations.
0 derived errors ignored.