求助，训练模型启动后报错

smdongxi1

运行训练SEAHEAD模型，然后大佬指导的炼丹步骤第一次预训练设置进行了设置，设置完成运行后，模型启动到80%就停了，然后弹出下面提示，请大佬指导，谢谢。


Initializing models:  80%|##################################################4            | 4/5 [05:54<01:28, 88.51s/it]
Error: OOM when allocating tensor with shape[524288,256] and type float on /job:localhost/replica:0/task:0/devICE:GPU:0 by allocator GPU_0_bfc
         [[node src_dst_opt/vs_inter_AB/dense1/weight_0/Assign (defined at D:\Tools\deepfacelab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:38) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.




Original stack trace for 'src_dst_opt/vs_inter_AB/dense1/weight_0/Assign':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\ModelBase.py", line 193, in __init__
    self.on_initialize()
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 341, in on_initialize
    self.src_dst_opt.initialize_variables (self.src_dst_saveable_weights, vars_on_cpu=optimizer_vars_on_cpu, lr_dropout_on_cpu=self.options['lr_dropout']=='cpu')
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in initialize_variables
    vs = { v.name : tf.get_variable ( f'vs_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in <dictcomp>
    vs = { v.name : tf.get_variable ( f'vs_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1595, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1338, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 593, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 545, in _true_getter
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 963, in _get_single_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 266, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 227, in _variable_v1_call
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 205, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2642, in default_variable_creator
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 270, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1670, in __init__
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1853, in _init_from_args
    validate_shape=validate_shape).op
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\state_ops.py", line 358, in assign
    validate_shape=validate_shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 59, in assign
    use_locking=use_locking, name=name)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal
    op_def=op_def)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)


Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
    return fn(*args)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn
    target_list, run_metadata)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[524288,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node src_dst_opt/vs_inter_AB/dense1/weight_0/Assign}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.




During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\ModelBase.py", line 193, in __init__
    self.on_initialize()
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 657, in on_initialize
    model.init_weights()
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\layers\Saveable.py", line 106, in init_weights
    nn.init_weights(self.get_weights())
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\ops\__init__.py", line 48, in init_weights
    nn.tf_sess.run (ops)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 968, in run
    run_metadata_ptr)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run
    feed_dict_tensor, options, run_metadata)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run
    run_metadata)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
    raise type(e)(node_def, op, message)  # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[524288,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node src_dst_opt/vs_inter_AB/dense1/weight_0/Assign (defined at D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py:38) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.




Original stack trace for 'src_dst_opt/vs_inter_AB/dense1/weight_0/Assign':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\ModelBase.py", line 193, in __init__
    self.on_initialize()
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 341, in on_initialize
    self.src_dst_opt.initialize_variables (self.src_dst_saveable_weights, vars_on_cpu=optimizer_vars_on_cpu, lr_dropout_on_cpu=self.options['lr_dropout']=='cpu')
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in initialize_variables
    vs = { v.name : tf.get_variable ( f'vs_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 38, in <dictcomp>
    vs = { v.name : tf.get_variable ( f'vs_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1595, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1338, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 593, in get_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 545, in _true_getter
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 963, in _get_single_variable
    aggregation=aggregation)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 266, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 227, in _variable_v1_call
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 205, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2642, in default_variable_creator
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 270, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1670, in __init__
    shape=shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1853, in _init_from_args
    validate_shape=validate_shape).op
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\state_ops.py", line 358, in assign
    validate_shape=validate_shape)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 59, in assign
    use_locking=use_locking, name=name)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal
    op_def=op_def)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)

番茄哥

oom爆显存了

五折

显存不够，降低BS

smdongxi1

收到，谢谢大佬们

smdongxi1

五折 发表于 2025-8-16 23:54
显存不够，降低BS

大佬，把BS降低到4了，还是一样啊，我显卡是笔记本的5060 8G的

smdongxi1

番茄哥 发表于 2025-8-16 22:40
oom爆显存了

大佬，把BS降低到4了，还是一样啊，我显卡是笔记本的5060 8G的

五折

smdongxi1 发表于 2025-8-20 11:10
大佬，把BS降低到4了，还是一样啊，我显卡是笔记本的5060 8G的

用的什么模型，8G能跑的模型并不多

smdongxi1

五折 发表于 2025-8-20 12:19
用的什么模型，8G能跑的模型并不多

DFL的SAEHD，用哪个更合适？

smdongxi1

五折 发表于 2025-8-20 12:19
用的什么模型，8G能跑的模型并不多

大佬，这次加载到100%了，但又报了其他错误
Initializing models: 100%|###############################################################| 5/5 [00:03<00:00,  1.38it/s]
Loading samples: 100%|##############################################################| 197/197 [00:01<00:00, 101.76it/s]
Loading samples: 100%|############################################################| 2795/2795 [00:17<00:00, 163.48it/s]
Process Process-21:
Process Process-20:
Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
    img = get_eyes_mouth_mask()*mask
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 79, in get_eyes_mouth_mask
    mask = eyes_mask + mouth_mask
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "multiprocessing\process.py", line 93, in run
Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
    gen_data = next (self.generator_func)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
    raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception: Exception occured in sample D:\Tools\DeepFaceLab-NV_RTX30_40_50\workspace\data_src\aligned\00032_0.jpg. Error: Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
    img = get_eyes_mouth_mask()*mask
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 79, in get_eyes_mouth_mask
    mask = eyes_mask + mouth_mask
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 113, in process
    warp_rnd_state=warp_rnd_state,
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\imagelib\warp.py", line 133, in gen_warp_params
    mapy = cv2.resize(mapy, (w+cell_size,)*2 )[half_cell_size:-half_cell_size,half_cell_size:-half_cell_size].astype(np.float32)
cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 1179648 bytes in function 'cv::OutOfMemoryError'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "multiprocessing\process.py", line 93, in run
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
    gen_data = next (self.generator_func)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
    raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception: Exception occured in sample D:\Tools\DeepFaceLab-NV_RTX30_40_50\workspace\data_src\aligned\00377_0.jpg. Error: Traceback (most recent call last):
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 113, in process
    warp_rnd_state=warp_rnd_state,
  File "D:\Tools\DeepFaceLab-NV_RTX30_40_50\_internal\DeepFaceLab\core\imagelib\warp.py", line 133, in gen_warp_params
    mapy = cv2.resize(mapy, (w+cell_size,)*2 )[half_cell_size:-half_cell_size,half_cell_size:-half_cell_size].astype(np.float32)
cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 1179648 bytes in function 'cv::OutOfMemoryError'

奸商

smdongxi1 发表于 2025-8-20 19:03
大佬，这次加载到100%了，但又报了其他错误
Initializing models: 100%|############################### ...

内存或虚拟内存不足
认真看教程