换脸时出问题

smdongxi1 · 发表于昨天 16:03

论坛购买了无敌小康的底丹，然后自己切脸训练，但在加载时弹出以下内容，不知道什么问题导致，还请各位大佬指点

使用设备: 0-NVIDIA GeForce RTX 5060 Laptop GPU (5.15gb/5.15gb)
ICE训练器: GPU 优化 RO 已关闭

装载类型:me-model: 100%|██████████████████████████| 8/8 [00:17<00:00,  2.23s/it]
装载训练样本: 100%|██████████████████████████| 151/151 [00:00<00:00, 442.53it/s]
装载训练样本: 100%|████████████████████████| 1373/1373 [00:02<00:00, 511.84it/s]

ICE 1.818 version by kingboy! QQ group:366893641
开始训练，当前迭代目标为：3000000，达到此目标数量将自动终结训练任务。

[当前时间][i:迭代数量][延迟ms]-[src损失][dst损失]
Traceback (most recent call last):
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
return fn(*args)
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1359, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1451, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[640,640,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
      [[{{node Conv2D_46}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

      [[gradients_1/Reshape_74_grad/Reshape/_521]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[640,640,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
      [[{{node Conv2D_46}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "models_ice\IceModel_Train.py", line 174, in models_ice.IceModel_Train.trainerThread
  File "models_ice\IceModel_Class.py", line 1025, in models_ice.IceModel_Class.IceModel_Class.train_one_iter
  File "models_ice\Ice_MULAN\Model.py", line 1580, in models_ice.Ice_MULAN.Model.MULANModel.onTrainOneIter
  File "models_ice\Ice_MULAN\Model.py", line 1269, in models_ice.Ice_MULAN.Model.MULANModel.on_init_model.train_sd
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "D:\Tools\FaceAI-ICE1.85\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[640,640,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
      [[node Conv2D_46 (defined at \threading.py:870) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

      [[gradients_1/Reshape_74_grad/Reshape/_521]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[640,640,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
      [[node Conv2D_46 (defined at \threading.py:870) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'Conv2D_46':
  File "\threading.py", line 890, in _bootstrap
self._bootstrap_inner()
  File "\threading.py", line 932, in _bootstrap_inner
self.run()
  File "\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
  File "\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
  File "\site-packages\tensorflow\python\ops\nn_ops.py", line 2269, in conv2d
return gen_nn_ops.conv2d(
  File "\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 968, in conv2d
_, _, _op, _outputs = _op_def_library._apply_op_helper(
  File "\site-packages\tensorflow\python\framework\op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
  File "\site-packages\tensorflow\python\framework\ops.py", line 3557, in _create_op_internal
ret = Operation(
  File "\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
self._traceback = tf_stack.extract_stack_for_node(self._c_op)