vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/modelarts/umn/modelarts_trouble_0054.html
Lai, Weijian 4e4b2d5f6d ModelArts UMN 23.3.0 Version. 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Lai, Weijian <laiweijian4@huawei.com>
Co-committed-by: Lai, Weijian <laiweijian4@huawei.com>
2024-06-26 07:03:02 +00:00

2.6 KiB
Raw Blame History

Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs

Symptom

When MindSpore is used for training, the following error message is displayed:
[ERROR] RUNTIME(3002)model execute error, retCode=0x91, [the model stream execute failed]

Possible Causes

The speed of reading data cannot keep up with the model iteration speed.

Solution

  1. Reduce shuffle operations during preprocessing.
    dataset = dataset.shuffle(buffer_size=x)
  2. Disable data preprocessing, which may affect system performance.
    NPURunConfig(enable_data_pre_proc=Flase)

Summary and Suggestions

Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.