kyxt
/
gpunpu
Not watched
1
0
Fork 1
Code
Releases 0 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree: d1096f376b
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'd1096f376b'
${ noResults }
gpunpu/wjtes2022092616t232775444-l...

233 lines
5.7 kB
Raw Blame History 

  1. /home/work

  2. 

  3. start loading script

  4. 

  5. finish loading script

  6. 

  7. 2022/09/26 16:24:20 Start to download master.zip 

  8. 

  9. 2022/09/26 16:24:20 Total parts count 1

  10. 

  11. 2022/09/26 16:24:21 part(1) finished

  12. 

  13. 2022/09/26 16:24:21 Download object finished, downloadPath:/cache/code/master.zip

  14. 

  15. panic: runtime error: index out of range [4] with length 4

  16. 

  17. 

  18. 

  19. goroutine 1 [running]:

  20. 

  21. main.main()

  22. 

  23. 	/home/houysh/openi/lewis/sync_for_grampus/downloader_for_obs.go:41 +0x4e0

  24. 

  25. unzip finished;start to exec code;

  26. 

  27. do nothing

  28. 

  29. [Modelarts Service Log]user: uid=1101(work) gid=1101(work) groups=1101(work),1000(HwHiAiUser)

  30. 

  31. [Modelarts Service Log]pwd: /home/work

  32. 

  33. [Modelarts Service Log]boot_file: /cache/code/npu_test/npu/train_for_c2net.py

  34. 

  35. [Modelarts Service Log]log_url: /tmp/log/train.log

  36. 

  37. [Modelarts Service Log]command: /cache/code/npu_test/npu/train_for_c2net.py

  38. 

  39. [Modelarts Service Log]local_code_dir: 

  40. 

  41. [Modelarts Service Log]Training start at 2022-09-26-16:24:21

  42. 

  43. [Modelarts Service Log][modelarts_create_log] modelarts-pipe found

  44. 

  45. [ModelArts Service Log]modelarts-pipe: will create log file /tmp/log/train.log

  46. 

  47. [Modelarts Service Log][modelarts_logger] modelarts-pipe found

  48. 

  49. [ModelArts Service Log]modelarts-pipe: will create log file /tmp/log/train.log

  50. 

  51. [ModelArts Service Log]modelarts-pipe: will write log file /tmp/log/train.log

  52. 

  53. [ModelArts Service Log]modelarts-pipe: param for max log length: 1073741824

  54. 

  55. [ModelArts Service Log]modelarts-pipe: param for whether exit on overflow: 0

  56. 

  57. INFO:root:Using MoXing-v2.0.0.rc2.4b57a67b-4b57a67b

  58. 

  59. INFO:root:Using OBS-Python-SDK-3.20.9.1

  60. 

  61. [Modelarts Service Log]2022-09-26 16:24:22,746 - INFO - Ascend Driver: Version=22.0.0.3

  62. 

  63. [Modelarts Service Log]2022-09-26 16:24:22,747 - INFO - you are advised to use ASCEND_DEVICE_ID env instead of DEVICE_ID, as the DEVICE_ID env will be discarded in later versions

  64. 

  65. [Modelarts Service Log]2022-09-26 16:24:22,747 - INFO - particularly, ${ASCEND_DEVICE_ID} == ${DEVICE_ID}, it's the logical device id

  66. 

  67. [Modelarts Service Log]2022-09-26 16:24:22,747 - INFO - Davinci training command

  68. 

  69. [Modelarts Service Log]2022-09-26 16:24:22,747 - INFO - ['/usr/bin/python', '/cache/code/npu_test/npu/train_for_c2net.py']

  70. 

  71. [Modelarts Service Log]2022-09-26 16:24:22,747 - INFO - Wait for Rank table file ready

  72. 

  73. [Modelarts Service Log]2022-09-26 16:24:22,748 - INFO - Rank table file (K8S generated) is ready for read

  74. 

  75. [Modelarts Service Log]2022-09-26 16:24:22,748 - INFO - 

  76. 

  77. {

  78. 

  79.     "status": "completed",

  80. 

  81.     "group_count": "1",

  82. 

  83.     "group_list": [

  84. 

  85.         {

  86. 

  87.             "group_name": "job-wjtes2022092616t2327",

  88. 

  89.             "device_count": "1",

  90. 

  91.             "instance_count": "1",

  92. 

  93.             "instance_list": [

  94. 

  95.                 {

  96. 

  97.                     "pod_name": "joba57ac677-job-wjtes2022092616t2327-0",

  98. 

  99.                     "server_id": "192.168.0.189",

  100. 

  101.                     "devices": [

  102. 

  103.                         {

  104. 

  105.                             "device_id": "3",

  106. 

  107.                             "device_ip": "192.4.68.236"

  108. 

  109.                         }

  110. 

  111.                     ]

  112. 

  113.                 }

  114. 

  115.             ]

  116. 

  117.         }

  118. 

  119.     ]

  120. 

  121. }

  122. 

  123. [Modelarts Service Log]2022-09-26 16:24:22,748 - INFO - Rank table file (C7x)

  124. 

  125. [Modelarts Service Log]2022-09-26 16:24:22,748 - INFO - 

  126. 

  127. {

  128. 

  129.     "status": "completed",

  130. 

  131.     "version": "1.0",

  132. 

  133.     "server_count": "1",

  134. 

  135.     "server_list": [

  136. 

  137.         {

  138. 

  139.             "server_id": "192.168.0.189",

  140. 

  141.             "device": [

  142. 

  143.                 {

  144. 

  145.                     "device_id": "3",

  146. 

  147.                     "device_ip": "192.4.68.236",

  148. 

  149.                     "rank_id": "0"

  150. 

  151.                 }

  152. 

  153.             ]

  154. 

  155.         }

  156. 

  157.     ]

  158. 

  159. }

  160. 

  161. [Modelarts Service Log]2022-09-26 16:24:22,749 - INFO - Rank table file (C7x) is generated

  162. 

  163. [Modelarts Service Log]2022-09-26 16:24:22,749 - INFO - Current server

  164. 

  165. [Modelarts Service Log]2022-09-26 16:24:22,749 - INFO - 

  166. 

  167. {

  168. 

  169.     "server_id": "192.168.0.189",

  170. 

  171.     "device": [

  172. 

  173.         {

  174. 

  175.             "device_id": "3",

  176. 

  177.             "device_ip": "192.4.68.236",

  178. 

  179.             "rank_id": "0"

  180. 

  181.         }

  182. 

  183.     ]

  184. 

  185. }

  186. 

  187. [Modelarts Service Log]2022-09-26 16:24:22,750 - INFO - bootstrap proc-rank-0-device-0

  188. 

  189. args:

  190. 

  191. Namespace(device_target='Ascend', epoch_size=5)

  192. 

  193. Traceback (most recent call last):

  194. 

  195.   File "/cache/code/npu_test/npu/train_for_c2net.py", line 50, in <module>

  196. 

  197.     cfg.batch_size)

  198. 

  199.   File "/cache/code/npu_test/npu/dataset.py", line 32, in create_dataset

  200. 

  201.     mnist_ds = ds.MnistDataset(data_path)

  202. 

  203.   File "/usr/local/ma/python3.7/lib/python3.7/site-packages/mindspore/dataset/engine/validators.py", line 343, in new_method

  204. 

  205.     check_dir(dataset_dir)

  206. 

  207.   File "/usr/local/ma/python3.7/lib/python3.7/site-packages/mindspore/dataset/core/validator_helpers.py", line 551, in check_dir

  208. 

  209.     raise ValueError("The folder {} does not exist or is not a directory or permission denied!".format(dataset_dir))

  210. 

  211. ValueError: The folder /cache/dataset/train does not exist or is not a directory or permission denied!

  212. 

  213. [Modelarts Service Log]2022-09-26 16:24:31,765 - ERROR - proc-rank-0-device-0 (pid: 159) has exited with non-zero code: 1

  214. 

  215. [Modelarts Service Log]2022-09-26 16:24:31,765 - INFO - Begin destroy training processes

  216. 

  217. [Modelarts Service Log]2022-09-26 16:24:31,765 - INFO - proc-rank-0-device-0 (pid: 159) has exited

  218. 

  219. [Modelarts Service Log]2022-09-26 16:24:31,765 - INFO - End destroy training processes

  220. 

  221. [ModelArts Service Log]modelarts-pipe: total length: 3763

  222. 

  223. [Modelarts Service Log]Training end with return code: 1

  224. 

  225. [Modelarts Service Log]Training end at 2022-09-26-16:24:31

  226. 

  227. [Modelarts Service Log]Training completed.

  228. 

  229. 2022/09/26 16:24:51 start uploading model

  230. 

  231. 2022/09/26 16:24:51 file:train.log

  232. 

  233. 2022/09/26 16:24:52 finish uploading model