Files

Lai, Weijian 6aa966a79a ModelArts UMN 24.3.0 version

Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Lai, Weijian <laiweijian4@huawei.com>
Co-committed-by: Lai, Weijian <laiweijian4@huawei.com>

2024-11-02 09:04:52 +00:00

2.3 KiB

Raw Permalink Blame History

Distributed Tensorflow Cannot Use tf.variable

Symptom

The following error occurs when tf.variable is used across multiple machines and multiple GPUs: WARNING:tensorflow:Gradient is None for variable:v0/tower_0/UNET_v7/sub_pixel/Variable:0.Make sure this variable is used in loss computation

Figure 1 Distributed Tensorflow unavailable

Possible Cause

Distributed TensorFlow needs to use tf.get_variable instead of tf.variable.

Solution

Replace tf.variable in the boot file with tf.get_variable.

Parent topic: Service Code Issues

2.3 KiB Raw Permalink Blame History

Distributed Tensorflow Cannot Use tf.variable

Symptom

Possible Cause

Solution

2.3 KiB

Raw Permalink Blame History