|
|
发表于 2024-5-21 20:28:19
|
显示全部楼层
设备序号: 0
设备名称: NVIDIA GeForce RTX 3070
显存大小: 5.33GB
____________________________________________________________
ICE 1.818 version by kingboy! QQ group:366893641
开始训练，预览窗口按 "H" 键获得功能帮助。
[当前时间][i:迭代数量][延迟ms]-[src损失][dst损失]
Traceback (most recent call last):
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
return fn(*args)
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1359, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1451, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[576,194,194] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Pad_17}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[concat_1/concat/_365]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[576,194,194] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Pad_17}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mainscripts\Trainer.py", line 164, in mainscripts.Trainer.trainerThread
File "models\ModelBase.py", line 573, in models.ModelBase.ModelBase.train_one_iter
File "models\Model_SAEHD\Model.py", line 868, in models.Model_SAEHD.Model.SAEHDModel.onTrainOneIter
File "models\Model_SAEHD\Model.py", line 633, in models.Model_SAEHD.Model.SAEHDModel.on_initialize.src_dst_train
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "D:\FaceAI-ICE1.85\FaceAI-ICE-1.818_fix2_2024_3.8\_internal\python-3.8.5\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[576,194,194] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node Pad_17 (defined at \threading.py:870) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[concat_1/concat/_365]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[576,194,194] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node Pad_17 (defined at \threading.py:870) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
Original stack trace for 'Pad_17':
File "\threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "\threading.py", line 932, in _bootstrap_inner
self.run()
File "\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
File "\site-packages\tensorflow\python\ops\array_ops.py", line 3514, in pad
result = gen_array_ops.pad(tensor, paddings, name=name)
File "\site-packages\tensorflow\python\ops\gen_array_ops.py", line 6503, in pad
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "\site-packages\tensorflow\python\framework\op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "\site-packages\tensorflow\python\framework\ops.py", line 3557, in _create_op_internal
ret = Operation(
File "\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
self._traceback = tf_stack.extract_stack_for_node(self._c_op) |
|
[复制链接]
照妖镜
楼主