!1036 UI add tensor error message

From: @xia_yi_fan1
Reviewed-by: @wenkai_dist,@lilongfei15
Signed-off-by: @lilongfei15

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "third_party/securec"]
 	path = third_party/securec
-	url = https://gitee.com/openeuler/bounds_checking_function.git
+	url = https://gitee.com/openeuler/libboundscheck.git

README.md

@@ -16,8 +16,8 @@ MindInsight provides MindSpore with easy-to-use debugging and tuning capabilitie
 
 ![MindInsight Architecture](docs/arch.png)
 
-Click to view the [Design document](https://www.mindspore.cn/doc/note/en/master/design.html) (visit [Design document](https://www.mindspore.cn/docs/en/master/design.html) before Sep. 24)，learn more about the design.
-Click to view the [Tutorial documentation](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html) (visit [Tutorial documentation](https://www.mindspore.cn/tutorial/en/master/advanced_use/visualization_tutorials.html) before Sep. 24) learn more about the MindInsight tutorial.
+Click to view the [Design document](https://www.mindspore.cn/doc/note/en/master/design/overall.html)，learn more about the design.
+Click to view the [Tutorial documentation](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html) learn more about the MindInsight tutorial.
 
 ## Installation
 Download whl package from [MindSpore download page](https://www.mindspore.cn/versions/en), and install the package.
@@ -31,7 +31,7 @@ For more details on how to install MindInsight, click on the MindInsight section
 ## Quick Start
 Before using MindInsight, the data in the training process should be recorded. When starting MindInsight, the directory of the saved data should be specified. After successful startup, the data can be viewed through the web page. Here is a brief introduction to recording training data, as well as starting and stopping MindInsight.
 
-[SummaryCollector](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector) (visit [SummaryCollector](https://www.mindspore.cn/api/en/master/api/python/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector) before Sep. 24) is the interface MindSpore provides for a quick and easy collection of common data about computational graphs, loss values, learning rates, parameter weights, and so on. Below is an example of using `SummaryCollector` for data collection, specifying the directory where the data is stored in `./summary_dir`.
+[SummaryCollector](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector) is the interface MindSpore provides for a quick and easy collection of common data about computational graphs, loss values, learning rates, parameter weights, and so on. Below is an example of using `SummaryCollector` for data collection, specifying the directory where the data is stored in `./summary_dir`.
 ```
 ...
 
@@ -40,7 +40,7 @@ summary_collector = SummaryCollector(summary_dir='./summary_dir')
 model.train(epoch=1, ds_train, callbacks=[summary_collector])
 ```
 
-For more ways to record visual data, see the [MindInsight Tutorial](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html) (visit [MindInsight Tutorial](https://www.mindspore.cn/tutorial/en/master/advanced_use/visualization_tutorials.html) before Sep. 24).
+For more ways to record visual data, see the [MindInsight Tutorial](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html).
 
 After you've collected the data, when you launch MindInsight, specify the directory in which the data has been stored.
 ```

README_CN.md

@@ -16,8 +16,8 @@ MindInsight为MindSpore提供了简单易用的调优调试能力。在训练过
 
 ![MindInsight Architecture](docs/arch.png)
 
-点击查看[设计文档](https://www.mindspore.cn/doc/note/zh-CN/master/design.html)（9月24日前请访问[设计文档](https://www.mindspore.cn/docs/zh-CN/master/design.html)），了解更多设计详情。
-点击查看[教程文档](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/visualization_tutorials.html)（9月24日前请访问[教程文档](https://www.mindspore.cn/tutorial/zh-CN/master/advanced_use/visualization_tutorials.html)），了解更多MindInsight教程。
+点击查看[设计文档](https://www.mindspore.cn/doc/note/zh-CN/master/design/overall.html)，了解更多设计详情。
+点击查看[教程文档](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/visualization_tutorials.html)，了解更多MindInsight教程。
 
 ## 安装
 请从[MindSpore下载页面](https://www.mindspore.cn/versions)下载并安装whl包。
@@ -32,7 +32,7 @@ pip install -U mindinsight-{version}-cp37-cp37m-linux_{arch}.whl
 使用MindInsight前，需要先将训练过程中的数据记录下来，启动MindInsight时，指定所保存的数据的位置，启动成功后，
 即可通过可视化页面查看数据。下面将简单介绍记录训练过程数据，以及启动、停止MindInsight服务。
 
-[SummaryCollector](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)（9月24日前请访问[SummaryCollector](https://www.mindspore.cn/api/zh-CN/master/api/python/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)）是MindSpore提供的快速简易地收集一些常见信息的接口，收集的信息包括计算图、损失值、学习率、参数权重等。
+[SummaryCollector](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)是MindSpore提供的快速简易地收集一些常见信息的接口，收集的信息包括计算图、损失值、学习率、参数权重等。
 下面是使用 `SummaryCollector` 进行数据收集的示例，其中指定存放数据的目录为 `./summary_dir`。
 ```
 ...
@@ -42,7 +42,7 @@ summary_collector = SummaryCollector(summary_dir='./summary_dir')
 model.train(epoch=1, ds_train, callbacks=[summary_collector])
 ```
 
-更多记录可视化数据的方法，请点击查看[MindInsight使用教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/visualization_tutorials.html)（9月24日前请访问[MindInsight使用教程](https://www.mindspore.cn/tutorial/zh-CN/master/advanced_use/visualization_tutorials.html)）。
+更多记录可视化数据的方法，请点击查看[MindInsight使用教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/visualization_tutorials.html)。
 
 收集好数据后，启动MindInsight时指定存放数据的目录。
 ```

RELEASE.md

@@ -1,5 +1,19 @@
 ## MindInsight
 
+# Release 1.0.1
+
+## Bugfixes
+* Update reference links of README and RELEASE. ([!731](https://gitee.com/mindspore/mindinsight/pulls/731/files))
+* Remove unused libs in requirements.txt. ([!745](https://gitee.com/mindspore/mindinsight/pulls/745/files))
+* Update securec repository link. ([!754](https://gitee.com/mindspore/mindinsight/pulls/754/files))
+
+## Thanks to our Contributors
+Thanks goes to these wonderful people:
+
+Congli Gao, Jianfeng Zhu, Zhenzhong Kou, Hongzhang Li, Longfei Li, Yongxiong Liang, Chongming Liu, Pengting Luo, Yanming Miao, Gongchang Ou, Yongxiu Qu, Luyu Qiu, Kai Wen, Yue Wang, Lihua Ye, Ximiao Yu, Yunshu Zhang, Ning Ma, Yihui Zhang, Shuide Wang, Hong Sheng, Ran Mo, Zhaohong Guo, Hui Pan, Junyan Qin, Weining Wang, Weifeng Huang, Yifan Xia.
+
+Contributions of any kind are welcome!
+
 # Release 1.0.0
 
 ## Major Features and Improvements
@@ -122,7 +136,7 @@ Contributions of any kind are welcome!
 
 ## Major Features and Improvements
 * Parameter distribution graph (Histogram).
-Now you can use [`HistogramSummary`](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.HistogramSummary) (visit [`HistogramSummary`](https://www.mindspore.cn/api/en/master/api/python/mindspore/mindspore.ops.html#mindspore.ops.HistogramSummary) before Sep. 24) and MindInsight to record and visualize distribution info of tensors. See our [tutorial](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html) (visit [tutorial](https://www.mindspore.cn/tutorial/en/master/advanced_use/visualization_tutorials.html) before Sep. 24) for details.
+Now you can use [`HistogramSummary`](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.HistogramSummary) and MindInsight to record and visualize distribution info of tensors. See our [tutorial](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/visualization_tutorials.html) for details.
 * Lineage support Custom information
 * GPU support
 * Model and dataset tracking linkage support

Third_Party_Open_Source_Software_Notice

@@ -0,0 +1,484 @@
+OPEN SOURCE SOFTWARE NOTICE
+
+Please note we provide an open source software notice along with this product and/or this product firmware (in the following just “this product”). The open source software licenses are granted by the respective right holders. And the open source licenses prevail all other license information with regard to the respective open source software contained in the product, including but not limited to End User Software Licensing Agreement. This notice is provided on behalf of Huawei Technologies Co. Ltd. and any of its local subsidiaries which may have provided this product to you in your local country. 
+
+Warranty Disclaimer    
+THE OPEN SOURCE SOFTWARE IN THIS PRODUCT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS.
+
+Copyright Notice and License Texts 
+
+Software: axios v0.18.1
+Copyright (c) 2014-present Matt Zabriskie
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: d3 v5.9.7
+Copyright 2010-2017 Mike Bostock
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the author nor the names of contributors may be used to
+  endorse or promote products derived from this software without specific prior
+  written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Software: d3-graphviz v3.0.4
+Copyright 2017, Magnus Jacobsson
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the author nor the names of contributors may be used to
+  endorse or promote products derived from this software without specific prior
+  written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Software: vue v2.6.11
+The MIT License (MIT)
+
+Copyright (c) 2013-present, Yuxi (Evan) You
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: vue-i18n v8.15.0
+The MIT License (MIT)
+
+Copyright (c) 2016 kazuya kawaguchi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: vuex v3.1.1
+The MIT License (MIT)
+
+Copyright (c) 2015-present Evan You
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: element-ui v2.11.1
+The MIT License (MIT)
+
+Copyright (c) 2016-present ElemeFE
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+Software: vue-router v3.1.3
+MIT License
+
+Copyright (c) 2013-present Evan You
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+Software: echarts v4.7.0
+The MIT License (MIT)
+
+Copyright (c) 2016-present GU Yiling & ECOMFE
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+Software: core-js v3.3.2
+Copyright (c) 2014-2020 Denis Pushkarev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: vue-i18n-loader v0.6.1
+The MIT License (MIT)
+
+Copyright (c) 2017 kazuya kawaguchi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: babel-eslint v10.0.3
+Copyright (c) 2014-2016 Sebastian McKenzie <sebmck@gmail.com>
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: eslint v6.6.0
+Copyright JS Foundation and other contributors, https://js.foundation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: eslint-plugin-vue v5.2.3
+MIT License
+
+Copyright (c) 2017 Toru Nagashima
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+Software: sass-loader v8.0.0
+Copyright JS Foundation and other contributors
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: dart-sass v1.25.0
+Copyright (c) 2016, Google Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: vue-cli v4.1.0
+The MIT License (MIT)
+
+Copyright (c) 2017-present, Yuxi (Evan) You
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+
+Software: vue-cli-plugin-i18n v0.6.1
+The MIT License (MIT)
+
+Copyright (c) 2018 kazuya kawaguchi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: slickGrid v2.4.22
+Copyright (c) 2009-2019 Michael Leibman and Ben McIntyre, http://github.com/6pac/slickgrid
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+Software: jquery v3.5.0
+Copyright JS Foundation and other contributors, https://js.foundation/
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

build/build.sh

@@ -28,8 +28,7 @@ rename_wheel() {
         PYTHON_VERSION_NUM=$(echo "$VERSION" | awk -F"." '{print $1$2}')
         PYTHON_VERSION_TAG="cp$PYTHON_VERSION_NUM"
         PYTHON_ABI_TAG="cp${PYTHON_VERSION_NUM}m"
-        OS_NAME=$(uname | tr '[:upper:]' '[:lower:]')
-        MACHINE_TAG="${OS_NAME}_$(uname -i)"
+        MACHINE_TAG="$(uname -s | tr '[:upper:]' '[:lower:]')_$(uname -m)"
         PACKAGE_NEW="mindinsight-$MINDINSIGHT_VERSION-$PYTHON_VERSION_TAG-$PYTHON_ABI_TAG-$MACHINE_TAG.whl"
         mv "$PACKAGE_ORIG" "$PACKAGE_NEW"
     done

build/scripts/ui.sh

@@ -28,7 +28,6 @@ build_ui() {
     rm -rf node_modules
     rm -rf dist
     mkdir -p public/static/js
-    cp ../../third_party/hpcc/graphvizlib.wasm public/static/js
 
     npm config set strict-ssl false
     npm config set unsafe-perm true
@@ -41,6 +40,8 @@ build_ui() {
         echo "dist does not have file index.html, build failed"
         exit 1
     fi
+
+    cp node_modules/@hpcc-js/wasm/dist/graphvizlib.wasm dist/static/js
 }
 
 build_ui

mindinsight/_version.py

@@ -14,4 +14,4 @@
 # ============================================================================
 """Mindinsight version module."""
 
-VERSION = '1.0.0'
+VERSION = '1.0.1'

mindinsight/backend/datavisual/static_resource_api.py

@@ -17,6 +17,7 @@ import os
 import sys
 
 from flask import current_app
+from flask import redirect
 from flask import send_from_directory
 from flask import Blueprint
 
@@ -34,6 +35,18 @@ def index():
     return send_from_directory(index_resource_dir, "index.html")
 
 
+@BLUEPRINT.route("/graphvizlib.wasm", methods=["GET"])
+def return_wasm_file():
+    """
+    Interface to redirect graphvizlib.wasm
+
+    When accessing the graphvizlib.wasm file in front module via Firefox browser, the file path will change to
+    "/graphvizlib.wasm" which makes the computed diagram inaccessible. Redirecting the path to correct address can
+    ensure the computed graph accessible properly.
+    """
+    return redirect(location="static/js/graphvizlib.wasm")
+
+
 def init_module(app):
     """
     Init module entry.

mindinsight/backend/lineagemgr/lineage_api.py

@@ -14,16 +14,15 @@
 # ============================================================================
 """Lineage restful api."""
 import json
-import os
 
 from flask import Blueprint, jsonify, request
 
 from mindinsight.conf import settings
 from mindinsight.datavisual.utils.tools import get_train_id
 from mindinsight.datavisual.data_transform.data_manager import DATA_MANAGER
-from mindinsight.lineagemgr.model import filter_summary_lineage, get_summary_lineage
-from mindinsight.utils.exceptions import MindInsightException, ParamValueError
 from mindinsight.lineagemgr.cache_item_updater import update_lineage_object
+from mindinsight.lineagemgr.model import filter_summary_lineage
+from mindinsight.utils.exceptions import MindInsightException, ParamValueError
 
 BLUEPRINT = Blueprint("lineage", __name__, url_prefix=settings.URL_PATH_PREFIX+settings.API_PREFIX)
 
@@ -67,26 +66,8 @@ def _get_lineage_info(search_condition):
     Raises:
         MindInsightException: If method fails to be called.
     """
-    summary_base_dir = str(settings.SUMMARY_BASE_DIR)
     try:
-        lineage_info = filter_summary_lineage(
-            data_manager=DATA_MANAGER,
-            search_condition=search_condition,
-            added=True)
-
-        lineages = lineage_info['object']
-
-        summary_base_dir = os.path.realpath(summary_base_dir)
-        length = len(summary_base_dir)
-
-        for lineage in lineages:
-            summary_dir = lineage['summary_dir']
-            summary_dir = os.path.realpath(summary_dir)
-            if summary_base_dir == summary_dir:
-                relative_dir = './'
-            else:
-                relative_dir = os.path.join(os.curdir, summary_dir[length+1:])
-            lineage['summary_dir'] = relative_dir
+        lineage_info = filter_summary_lineage(data_manager=DATA_MANAGER, search_condition=search_condition)
 
     except MindInsightException as exception:
         raise MindInsightException(exception.error, exception.message, http_code=400)
@@ -134,29 +115,27 @@ def get_dataset_graph():
         >>> GET http://xxxx/v1/mindinsight/datasets/dataset_graph?train_id=xxx
     """
 
-    summary_base_dir = str(settings.SUMMARY_BASE_DIR)
-    summary_dir = get_train_id(request)
+    train_id = get_train_id(request)
+    search_condition = {
+        'summary_dir': {
+            'in': [train_id]
+        }
+    }
+    result = {}
     try:
-        dataset_graph = get_summary_lineage(
-            DATA_MANAGER,
-            summary_dir=summary_dir,
-            keys=['dataset_graph']
-        )
+        objects = filter_summary_lineage(DATA_MANAGER, search_condition).get('object')
     except MindInsightException as exception:
         raise MindInsightException(exception.error, exception.message, http_code=400)
 
-    if dataset_graph:
-        summary_dir_result = dataset_graph.get('summary_dir')
-        base_dir_len = len(summary_base_dir)
-        if summary_base_dir == summary_dir_result:
-            relative_dir = './'
-        else:
-            relative_dir = os.path.join(
-                os.curdir, summary_dir[base_dir_len + 1:]
-            )
-        dataset_graph['summary_dir'] = relative_dir
-
-    return jsonify(dataset_graph)
+    if objects:
+        lineage_obj = objects[0]
+        dataset_graph = lineage_obj.get('dataset_graph')
+
+        if dataset_graph:
+            result.update({'dataset_graph': dataset_graph})
+            result.update({'summary_dir': lineage_obj.get('summary_dir')})
+
+    return jsonify(result)
 
 
 def init_module(app):

mindinsight/datavisual/common/exceptions.py

@@ -195,3 +195,12 @@ class TrainJobDetailNotInCacheError(MindInsightException):
         super().__init__(DataVisualErrors.TRAIN_JOB_DETAIL_NOT_IN_CACHE,
                          error_msg,
                          http_code=400)
+
+
+class TensorTooLargeError(MindInsightException):
+    """The given tensor is too large to shown on UI."""
+    def __init__(self, error_detail):
+        error_msg = f'Tensor is too large to show on UI. Detail: {error_detail}'
+        super(TensorTooLargeError, self).__init__(DataVisualErrors.TENSOR_TOO_LARGE,
+                                                  error_msg,
+                                                  http_code=400)

mindinsight/datavisual/data_transform/data_manager.py

@@ -210,6 +210,11 @@ class CachedTrainJob:
         """Threading lock with given key."""
         return self._key_locks.setdefault(key, threading.Lock())
 
+    @property
+    def train_id(self):
+        """Get train id."""
+        return self._basic_info.train_id
+
 
 class TrainJob:
     """

mindinsight/datavisual/data_transform/ms_data_loader.py

@@ -483,10 +483,9 @@ class _SummaryParser(_Parser):
 
         elif plugin == PluginNameEnum.TENSOR.value:
             tensor_event_value = TensorContainer(tensor_event_value)
-            if tensor_event_value.size > MAX_TENSOR_COUNT:
+            if tensor_event_value.error_code is not None:
                 logger.warning('tag: %s/tensor, dims: %s, tensor count: %d exceeds %d and drop it.',
                                value.tag, tensor_event_value.dims, tensor_event_value.size, MAX_TENSOR_COUNT)
-                return None
 
         elif plugin == PluginNameEnum.IMAGE.value:
             tensor_event_value = ImageContainer(tensor_event_value)

mindinsight/datavisual/data_transform/tensor_container.py

@@ -17,10 +17,12 @@ import numpy as np
 
 from mindinsight.datavisual.data_transform.histogram import Histogram, Bucket
 from mindinsight.datavisual.utils.utils import calc_histogram_bins
+from mindinsight.datavisual.common.exceptions import TensorTooLargeError
 from mindinsight.utils.exceptions import ParamValueError
 from mindinsight.utils.tensor import TensorUtils
 
 MAX_TENSOR_COUNT = 10000000
+TENSOR_TOO_LARGE_ERROR = TensorTooLargeError("").error_code
 
 
 def calc_original_buckets(np_value, stats):
@@ -74,6 +76,10 @@ class TensorContainer:
         self._dims = tuple(tensor_message.dims)
         self._data_type = tensor_message.data_type
         self._np_array = self.get_ndarray(tensor_message.float_data)
+        self._error_code = None
+        if self._np_array.size > MAX_TENSOR_COUNT:
+            self._error_code = TENSOR_TOO_LARGE_ERROR
+            self._np_array = np.array([])
         self._stats = TensorUtils.get_statistics_from_tensor(self._np_array)
         original_buckets = calc_original_buckets(self._np_array, self._stats)
         self._count = sum(bucket.count for bucket in original_buckets)
@@ -86,6 +92,11 @@ class TensorContainer:
         """Get size of tensor."""
         return self._np_array.size
 
+    @property
+    def error_code(self):
+        """Get size of tensor."""
+        return self._error_code
+
     @property
     def dims(self):
         """Get dims of tensor."""
@@ -128,6 +139,8 @@ class TensorContainer:
 
     def buckets(self):
         """Get histogram buckets."""
+        if self._histogram is None:
+            return None
         return self._histogram.buckets()
 
     def get_ndarray(self, tensor):

mindinsight/datavisual/processors/tensor_processor.py

@@ -23,7 +23,7 @@ from mindinsight.utils.tensor import TensorUtils, MAX_DIMENSIONS_FOR_TENSOR
 from mindinsight.conf.constants import MAX_TENSOR_RESPONSE_DATA_SIZE
 from mindinsight.datavisual.common.validation import Validation
 from mindinsight.datavisual.common.exceptions import StepTensorDataNotInCacheError, TensorNotExistError
-from mindinsight.datavisual.common.exceptions import ResponseDataExceedMaxValueError
+from mindinsight.datavisual.common.exceptions import ResponseDataExceedMaxValueError, TensorTooLargeError
 from mindinsight.datavisual.data_transform.tensor_container import TensorContainer
 from mindinsight.datavisual.processors.base_processor import BaseProcessor
 from mindinsight.datavisual.proto_files import mindinsight_anf_ir_pb2 as anf_ir_pb2
@@ -153,7 +153,9 @@ class TensorProcessor(BaseProcessor):
                 "data_type": anf_ir_pb2.DataType.Name(value.data_type)
             }
             if detail and detail == 'stats':
-                stats = TensorUtils.get_statistics_dict(stats=value.stats, overall_stats=value.stats)
+                stats = None
+                if value.error_code is None:
+                    stats = TensorUtils.get_statistics_dict(stats=value.stats, overall_stats=value.stats)
                 value_dict.update({"statistics": stats})
 
             values.append({
@@ -206,6 +208,8 @@ class TensorProcessor(BaseProcessor):
             if step != tensor.step:
                 continue
             step_in_cache = True
+            if value.error_code is not None:
+                raise TensorTooLargeError("Step: {}".format(tensor.step))
             res_data = TensorUtils.get_specific_dims_data(value.ndarray, dims)
             flatten_data = res_data.flatten().tolist()
             if len(flatten_data) > MAX_TENSOR_RESPONSE_DATA_SIZE:
@@ -285,6 +289,8 @@ class TensorProcessor(BaseProcessor):
         for tensor in tensors:
             # This value is an instance of TensorContainer
             value = tensor.value
+            if value.error_code is not None:
+                raise TensorTooLargeError("Step: {}".format(tensor.step))
             buckets = value.buckets()
             values.append({
                 "wall_time": tensor.wall_time,

mindinsight/lineagemgr/cache_item_updater.py

@@ -72,12 +72,13 @@ class LineageCacheItemUpdater(BaseCacheItemUpdater):
 
     def _lineage_parsing(self, cache_item):
         """Parse summaries and return lineage parser."""
+        train_id = cache_item.train_id
         summary_dir = cache_item.abs_summary_dir
         update_time = cache_item.basic_info.update_time
 
         cached_lineage_item = cache_item.get(key=LINEAGE, raise_exception=False)
         if cached_lineage_item is None:
-            lineage_parser = LineageParser(summary_dir, update_time)
+            lineage_parser = LineageParser(train_id, summary_dir, update_time)
         else:
             lineage_parser = cached_lineage_item
             with cache_item.lock_key(LINEAGE):

mindinsight/lineagemgr/common/exceptions/error_code.py

@@ -57,7 +57,6 @@ class LineageErrors(LineageErrorCodes):
     PARAM_STEP_NUM_ERROR = 18 | _MINDSPORE_COLLECTOR_ERROR
 
     # Model lineage error codes.
-    LINEAGE_PARAM_OPERATION_ERROR = 0 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_PARAM_METRIC_ERROR = 1 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_PARAM_LOSS_FUNCTION_ERROR = 4 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_PARAM_TRAIN_DATASET_PATH_ERROR = 5 | _MODEL_LINEAGE_API_ERROR_MASK
@@ -79,7 +78,6 @@ class LineageErrors(LineageErrorCodes):
     LINEAGE_DIR_NOT_EXIST_ERROR = 20 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_SUMMARY_DATA_ERROR = 21 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_FILE_NOT_FOUND_ERROR = 22 | _MODEL_LINEAGE_API_ERROR_MASK
-    LINEAGE_PARAM_SUMMARY_PATH_ERROR = 23 | _MODEL_LINEAGE_API_ERROR_MASK
     LINEAGE_SEARCH_CONDITION_PARAM_ERROR = 24 | _MODEL_LINEAGE_API_ERROR_MASK
 
     LINEAGE_PARAM_LINEAGE_TYPE_ERROR = 25 | _MODEL_LINEAGE_API_ERROR_MASK
@@ -110,7 +108,6 @@ class LineageErrorMsg(Enum):
                                  "mindspore.train.summary.SummaryRecord"
     PARAM_RAISE_EXCEPTION_ERROR = "Invalid value for raise_exception. It should be True or False."
     # Lineage error messages.
-    LINEAGE_PARAM_SUMMARY_PATH_ERROR = "The parameter summary path error: {}"
     LINEAGE_SUMMARY_DATA_ERROR = "Query summary data error: {}"
     LINEAGE_FILE_NOT_FOUND_ERROR = "File not found error: {}"
     LINEAGE_DIR_NOT_EXIST_ERROR = "Dir not exist error: {}"

mindinsight/lineagemgr/common/exceptions/exceptions.py

@@ -106,16 +106,6 @@ class LineageEventFieldNotExistException(MindInsightException):
         )
 
 
-class LineageParamSummaryPathError(MindInsightException):
-    """The lineage parameter summary path error."""
-    def __init__(self, msg):
-        super(LineageParamSummaryPathError, self).__init__(
-            error=LineageErrors.LINEAGE_PARAM_SUMMARY_PATH_ERROR,
-            message=LineageErrorMsg.LINEAGE_PARAM_SUMMARY_PATH_ERROR.value.format(msg),
-            http_code=400
-        )
-
-
 class LineageQuerySummaryDataError(MindInsightException):
     """Query summary data error in lineage module."""
     def __init__(self, msg):
@@ -136,16 +126,6 @@ class LineageFileNotFoundError(MindInsightException):
         )
 
 
-class LineageDirNotExistError(MindInsightException):
-    """Directory not exist in lineage module."""
-    def __init__(self, msg):
-        super(LineageDirNotExistError, self).__init__(
-            error=LineageErrors.LINEAGE_DIR_NOT_EXIST_ERROR,
-            message=LineageErrorMsg.LINEAGE_DIR_NOT_EXIST_ERROR.value.format(msg),
-            http_code=400
-        )
-
-
 class LineageSearchConditionParamError(MindInsightException):
     """Search condition param is invalid in lineage module."""
     def __init__(self, msg):

mindinsight/lineagemgr/common/utils.py

@@ -15,30 +15,16 @@
 """Lineage utils."""
 import os
 import re
-from pathlib import Path
 
 from mindinsight.datavisual.data_transform.summary_watcher import SummaryWatcher
-from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamValueError, LineageParamTypeError, \
-    LineageDirNotExistError, LineageParamSummaryPathError
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamTypeError
 from mindinsight.lineagemgr.common.log import logger as log
-from mindinsight.lineagemgr.common.validator.validate import validate_path
 
 
 def enum_to_list(enum):
     return [enum_ele.value for enum_ele in enum]
 
 
-def normalize_summary_dir(summary_dir):
-    """Normalize summary dir."""
-    try:
-        summary_dir = validate_path(summary_dir)
-    except (LineageParamValueError, LineageDirNotExistError) as error:
-        log.error(str(error))
-        log.exception(error)
-        raise LineageParamSummaryPathError(str(error.message))
-    return summary_dir
-
-
 def get_timestamp(filename):
     """Get timestamp from filename."""
     timestamp = int(re.search(SummaryWatcher().SUMMARY_FILENAME_REGEX, filename)[1])
@@ -68,25 +54,3 @@ def make_directory(path):
             log.error("No write permission on the directory(%r), error = %r", path, err)
             raise LineageParamTypeError("No write permission on the directory.")
     return real_path
-
-
-def get_relative_path(path, base_path):
-    """
-    Get relative path based on base_path.
-
-    Args:
-        path (str): absolute path.
-        base_path: absolute base path.
-
-    Returns:
-        str, relative path based on base_path.
-
-    """
-    try:
-        r_path = str(Path(path).relative_to(Path(base_path)))
-    except ValueError:
-        raise LineageParamValueError("The path %r does not start with %r." % (path, base_path))
-
-    if r_path == ".":
-        r_path = ""
-    return os.path.join("./", r_path)

mindinsight/lineagemgr/common/validator/validate.py

@@ -13,13 +13,11 @@
 # limitations under the License.
 # ============================================================================
 """Validate the parameters."""
-import os
 import re
 from marshmallow import ValidationError
 
 from mindinsight.lineagemgr.common.exceptions.error_code import LineageErrors, LineageErrorMsg
-from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamTypeError, \
-    LineageParamValueError, LineageDirNotExistError
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamTypeError, LineageParamValueError
 from mindinsight.lineagemgr.common.log import logger as log
 from mindinsight.lineagemgr.common.validator.validate_path import safe_normalize_path
 from mindinsight.lineagemgr.querier.query_model import FIELD_MAPPING
@@ -224,38 +222,6 @@ def validate_raise_exception(raise_exception):
         )
 
 
-def validate_filter_key(keys):
-    """
-    Verify the keys of filtering is valid or not.
-
-    Args:
-        keys (list): The keys to get the relative lineage info.
-
-    Raises:
-        LineageParamTypeError: If keys is not list.
-        LineageParamValueError: If the value of keys is invalid.
-    """
-    filter_keys = [
-        'metric', 'hyper_parameters', 'algorithm',
-        'train_dataset', 'model', 'valid_dataset',
-        'dataset_graph'
-    ]
-
-    if not isinstance(keys, list):
-        log.error("Keys must be list.")
-        raise LineageParamTypeError("Keys must be list.")
-
-    for element in keys:
-        if not isinstance(element, str):
-            log.error("Element of keys must be str.")
-            raise LineageParamTypeError("Element of keys must be str.")
-
-    if not set(keys).issubset(filter_keys):
-        err_msg = "Keys must be in {}.".format(filter_keys)
-        log.error(err_msg)
-        raise LineageParamValueError(err_msg)
-
-
 def validate_condition(search_condition):
     """
     Verify the param in search_condition is valid or not.
@@ -310,31 +276,6 @@ def validate_condition(search_condition):
             raise LineageParamValueError(err_msg)
 
 
-def validate_path(summary_path):
-    """
-    Verify the summary path is valid or not.
-
-    Args:
-        summary_path (str): The summary path which is a dir.
-
-    Raises:
-        LineageParamValueError: If the input param value is invalid.
-        LineageDirNotExistError: If the summary path is invalid.
-    """
-    try:
-        summary_path = safe_normalize_path(
-            summary_path, "summary_path", None, check_absolute_path=True
-        )
-    except ValidationError:
-        log.error("The summary path is invalid.")
-        raise LineageParamValueError("The summary path is invalid.")
-    if not os.path.isdir(summary_path):
-        log.error("The summary path does not exist or is not a dir.")
-        raise LineageDirNotExistError("The summary path does not exist or is not a dir.")
-
-    return summary_path
-
-
 def validate_user_defined_info(user_defined_info):
     """
     Validate user defined info， delete the item if its key is in lineage.

mindinsight/lineagemgr/lineage_parser.py

@@ -15,7 +15,6 @@
 """This file is used to parse lineage info."""
 import os
 
-from mindinsight.datavisual.data_transform.summary_watcher import SummaryWatcher
 from mindinsight.lineagemgr.common.exceptions.exceptions import LineageSummaryAnalyzeException, \
     LineageEventNotExistException, LineageEventFieldNotExistException, LineageFileNotFoundError, \
     MindInsightException
@@ -64,8 +63,9 @@ class SuperLineageObj:
 
 class LineageParser:
     """Lineage parser."""
-    def __init__(self, summary_dir, update_time=None, added_info=None):
+    def __init__(self, train_id, summary_dir, update_time=None, added_info=None):
         self._summary_dir = summary_dir
+        self._train_id = train_id
         self._update_time = update_time
         self._added_info = added_info
 
@@ -154,7 +154,7 @@ class LineageParser:
         """Update lineage object."""
         if self._super_lineage_obj is None:
             lineage_obj = LineageObj(
-                self._summary_dir,
+                self._train_id,
                 train_lineage=lineage_info.train_lineage,
                 evaluation_lineage=lineage_info.eval_lineage,
                 dataset_graph=lineage_info.dataset_graph,
@@ -177,52 +177,13 @@ class LineageParser:
 
 class LineageOrganizer:
     """Lineage organizer."""
-    def __init__(self, data_manager=None, summary_base_dir=None):
+    def __init__(self, data_manager):
         self._data_manager = data_manager
-        self._summary_base_dir = summary_base_dir
-        self._check_params()
         self._super_lineage_objs = {}
         self._organize_from_cache()
-        self._organize_from_disk()
-
-    def _check_params(self):
-        """Check params."""
-        if self._data_manager is not None and self._summary_base_dir is not None:
-            self._summary_base_dir = None
-
-    def _organize_from_disk(self):
-        """Organize lineage objs from disk."""
-        if self._summary_base_dir is None:
-            return
-        summary_watcher = SummaryWatcher()
-        relative_dirs = summary_watcher.list_summary_directories(
-            summary_base_dir=self._summary_base_dir
-        )
-
-        no_lineage_count = 0
-        for item in relative_dirs:
-            relative_dir = item.get('relative_path')
-            update_time = item.get('update_time')
-            abs_summary_dir = os.path.realpath(os.path.join(self._summary_base_dir, relative_dir))
-
-            try:
-                lineage_parser = LineageParser(abs_summary_dir, update_time)
-                super_lineage_obj = lineage_parser.super_lineage_obj
-                if super_lineage_obj is not None:
-                    self._super_lineage_objs.update({abs_summary_dir: super_lineage_obj})
-            except LineageFileNotFoundError:
-                no_lineage_count += 1
-
-        if no_lineage_count == len(relative_dirs):
-            logger.info('There is no summary log file under summary_base_dir.')
-            raise LineageFileNotFoundError(
-                'There is no summary log file under summary_base_dir.'
-            )
 
     def _organize_from_cache(self):
         """Organize lineage objs from cache."""
-        if self._data_manager is None:
-            return
         brief_cache = self._data_manager.get_brief_cache()
         cache_items = brief_cache.cache_items
         for relative_dir, cache_train_job in cache_items.items():

mindinsight/lineagemgr/model.py

@@ -13,20 +13,15 @@
 # limitations under the License.
 # ============================================================================
 """This file is used to define the model lineage python api."""
-import os
 import numpy as np
 import pandas as pd
 
-from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamValueError, \
-    LineageQuerySummaryDataError, LineageParamSummaryPathError, \
-    LineageQuerierParamException, LineageDirNotExistError, LineageSearchConditionParamError, \
-    LineageParamTypeError, LineageSummaryParseException
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageQuerySummaryDataError, \
+    LineageQuerierParamException, LineageSearchConditionParamError, LineageParamTypeError, LineageSummaryParseException
 from mindinsight.lineagemgr.common.log import logger as log
-from mindinsight.lineagemgr.common.utils import normalize_summary_dir, get_relative_path
 from mindinsight.lineagemgr.common.validator.model_parameter import SearchModelConditionParameter
-from mindinsight.lineagemgr.common.validator.validate import validate_filter_key, validate_search_model_condition, \
-    validate_condition, validate_path, validate_train_id
-from mindinsight.lineagemgr.lineage_parser import LineageParser, LineageOrganizer
+from mindinsight.lineagemgr.common.validator.validate import validate_search_model_condition, validate_condition
+from mindinsight.lineagemgr.lineage_parser import LineageOrganizer
 from mindinsight.lineagemgr.querier.querier import Querier
 from mindinsight.optimizer.common.enums import ReasonCode
 from mindinsight.optimizer.utils.utils import is_simple_numpy_number
@@ -38,62 +33,7 @@ _USER_DEFINED_PREFIX = "[U]"
 USER_DEFINED_INFO_LIMIT = 100
 
 
-def get_summary_lineage(data_manager=None, summary_dir=None, keys=None):
-    """
-    Get summary lineage from data_manager or parsing from summaries.
-
-    One of data_manager or summary_dir needs to be specified. Support getting
-    super_lineage_obj from data_manager or parsing summaries by summary_dir.
-
-    Args:
-        data_manager (DataManager): Data manager defined as
-            mindinsight.datavisual.data_transform.data_manager.DataManager
-        summary_dir (str): The summary directory. It contains summary logs for
-            one training.
-        keys (list[str]): The filter keys of lineage information. The acceptable
-            keys are `metric`, `user_defined`, `hyper_parameters`, `algorithm`,
-            `train_dataset`, `model`, `valid_dataset` and `dataset_graph`.
-            If it is `None`, all information will be returned. Default: None.
-
-    Returns:
-        dict, the lineage information for one training.
-
-    Raises:
-        LineageParamSummaryPathError: If summary path is invalid.
-        LineageQuerySummaryDataError: If querying summary data fails.
-        LineageFileNotFoundError: If the summary log file is not found.
-
-    """
-    default_result = {}
-    if data_manager is None and summary_dir is None:
-        raise LineageParamTypeError("One of data_manager or summary_dir needs to be specified.")
-    if data_manager is not None and summary_dir is None:
-        raise LineageParamTypeError("If data_manager is specified, the summary_dir needs to be "
-                                    "specified as relative path.")
-
-    if keys is not None:
-        validate_filter_key(keys)
-
-    if data_manager is None:
-        normalize_summary_dir(summary_dir)
-        super_lineage_obj = LineageParser(summary_dir).super_lineage_obj
-    else:
-        validate_train_id(summary_dir)
-        super_lineage_obj = LineageOrganizer(data_manager=data_manager).get_super_lineage_obj(summary_dir)
-
-    if super_lineage_obj is None:
-        return default_result
-
-    try:
-        result = Querier({summary_dir: super_lineage_obj}).get_summary_lineage(summary_dir, keys)
-    except (LineageQuerierParamException, LineageParamTypeError) as error:
-        log.error(str(error))
-        log.exception(error)
-        raise LineageQuerySummaryDataError("Get summary lineage failed.")
-    return result[0]
-
-
-def filter_summary_lineage(data_manager=None, summary_base_dir=None, search_condition=None, added=False):
+def filter_summary_lineage(data_manager, search_condition=None):
     """
     Filter summary lineage from data_manager or parsing from summaries.
 
@@ -103,18 +43,8 @@ def filter_summary_lineage(data_manager=None, summary_base_dir=None, search_cond
     Args:
         data_manager (DataManager): Data manager defined as
             mindinsight.datavisual.data_transform.data_manager.DataManager
-        summary_base_dir (str): The summary base directory. It contains summary
-            directories generated by training.
         search_condition (dict): The search condition.
     """
-    if data_manager is None and summary_base_dir is None:
-        raise LineageParamTypeError("One of data_manager or summary_base_dir needs to be specified.")
-
-    if data_manager is None:
-        summary_base_dir = normalize_summary_dir(summary_base_dir)
-    else:
-        summary_base_dir = data_manager.summary_base_dir
-
     search_condition = {} if search_condition is None else search_condition
 
     try:
@@ -126,18 +56,8 @@ def filter_summary_lineage(data_manager=None, summary_base_dir=None, search_cond
         raise LineageSearchConditionParamError(str(error.message))
 
     try:
-        search_condition = _convert_relative_path_to_abspath(summary_base_dir, search_condition)
-    except (LineageParamValueError, LineageDirNotExistError) as error:
-        log.error(str(error))
-        log.exception(error)
-        raise LineageParamSummaryPathError(str(error.message))
-
-    try:
-        lineage_objects = LineageOrganizer(data_manager, summary_base_dir).super_lineage_objs
-        result = Querier(lineage_objects).filter_summary_lineage(
-            condition=search_condition,
-            added=added
-        )
+        lineage_objects = LineageOrganizer(data_manager).super_lineage_objs
+        result = Querier(lineage_objects).filter_summary_lineage(condition=search_condition)
     except LineageSummaryParseException:
         result = {'object': [], 'count': 0}
     except (LineageQuerierParamException, LineageParamTypeError) as error:
@@ -148,53 +68,6 @@ def filter_summary_lineage(data_manager=None, summary_base_dir=None, search_cond
     return result
 
 
-def _convert_relative_path_to_abspath(summary_base_dir, search_condition):
-    """
-    Convert relative path to absolute path.
-
-    Args:
-        summary_base_dir (str): The summary base directory.
-        search_condition (dict): The search condition.
-
-    Returns:
-        dict, the updated search_condition.
-
-    Raises:
-        LineageParamValueError: If the value of input_name is invalid.
-    """
-    if ("summary_dir" not in search_condition) or (not search_condition.get("summary_dir")):
-        return search_condition
-
-    summary_dir_condition = search_condition.get("summary_dir")
-
-    for key in ['in', 'not_in']:
-        if key in summary_dir_condition:
-            summary_paths = []
-            for summary_dir in summary_dir_condition.get(key):
-                if summary_dir.startswith('./'):
-                    abs_dir = os.path.join(
-                        summary_base_dir, summary_dir[2:]
-                    )
-                    abs_dir = validate_path(abs_dir)
-                else:
-                    abs_dir = validate_path(summary_dir)
-                summary_paths.append(abs_dir)
-            search_condition.get('summary_dir')[key] = summary_paths
-
-    if 'eq' in summary_dir_condition:
-        summary_dir = summary_dir_condition.get('eq')
-        if summary_dir.startswith('./'):
-            abs_dir = os.path.join(
-                summary_base_dir, summary_dir[2:]
-            )
-            abs_dir = validate_path(abs_dir)
-        else:
-            abs_dir = validate_path(summary_dir)
-        search_condition.get('summary_dir')['eq'] = abs_dir
-
-    return search_condition
-
-
 def get_flattened_lineage(data_manager, search_condition=None):
     """
     Get lineage data in a table from data manager.
@@ -207,10 +80,10 @@ def get_flattened_lineage(data_manager, search_condition=None):
         Dict[str, list]: A dict contains keys and values from lineages.
 
     """
-    summary_base_dir, flatten_dict, user_count = data_manager.summary_base_dir, {'train_id': []}, 0
+    flatten_dict, user_count = {'train_id': []}, 0
     lineages = filter_summary_lineage(data_manager=data_manager, search_condition=search_condition).get("object", [])
     for index, lineage in enumerate(lineages):
-        flatten_dict['train_id'].append(get_relative_path(lineage.get("summary_dir"), summary_base_dir))
+        flatten_dict['train_id'].append(lineage.get("summary_dir"))
         for key, val in _flatten_lineage(lineage.get('model_lineage', {})):
             if key.startswith(_USER_DEFINED_PREFIX) and key not in flatten_dict:
                 if user_count > USER_DEFINED_INFO_LIMIT:

mindinsight/lineagemgr/querier/querier.py

@@ -189,53 +189,7 @@ class Querier:
             raise LineageParamTypeError("Init param should be a dict.")
         return super_lineage_objs
 
-    def get_summary_lineage(self, summary_dir=None, filter_keys=None):
-        """
-        Get summary lineage information.
-
-        If a summary dir is specified, the special summary lineage information
-        will be found. If the summary dir is `None`, all summary lineage
-        information will be found.
-
-        Returns the content corresponding to the specified field in the filter
-        key. The contents of the filter key include `metric`, `hyper_parameters`,
-        `algorithm`, `train_dataset`, `valid_dataset` and `model`. You can
-        specify multiple filter keys in the `filter_keys`. If the parameter is
-        `None`, complete information will be returned.
-
-        Args:
-            summary_dir (Union[str, None]): Summary log dir. Default: None.
-            filter_keys (Union[list[str], None]): Filter keys. Default: None.
-
-        Returns:
-            list[dict], summary lineage information.
-        """
-
-        if filter_keys is None:
-            filter_keys = LineageFilterKey.get_key_list()
-        else:
-            for key in filter_keys:
-                if not LineageFilterKey.is_valid_filter_key(key):
-                    raise LineageQuerierParamException(
-                        filter_keys, 'The filter key {} is invalid.'.format(key)
-                    )
-
-        if summary_dir is None:
-            result = [
-                item.lineage_obj.get_summary_info(filter_keys) for item in self._super_lineage_objs.values()
-            ]
-        elif summary_dir in self._super_lineage_objs:
-            lineage_obj = self._super_lineage_objs[summary_dir].lineage_obj
-            result = [lineage_obj.get_summary_info(filter_keys)]
-        else:
-            raise LineageQuerierParamException(
-                'summary_dir',
-                'Summary dir {} does not exist.'.format(summary_dir)
-            )
-
-        return result
-
-    def filter_summary_lineage(self, condition=None, added=False):
+    def filter_summary_lineage(self, condition=None):
         """
         Filter and sort lineage information based on the specified condition.
 
@@ -298,8 +252,7 @@ class Querier:
                 lineage_object.update(item.lineage_obj.to_model_lineage_dict())
             if LineageType.DATASET.value in lineage_types:
                 lineage_object.update(item.lineage_obj.to_dataset_lineage_dict())
-            if added:
-                lineage_object.update({"added_info": item.added_info})
+            lineage_object.update({"added_info": item.added_info})
             object_items.append(lineage_object)
 
         lineage_info = {

mindinsight/mindconverter/README.md

@@ -124,7 +124,7 @@ In the conversion report, non-transformed code is listed as follows:
 line <row>:<col> [UnConvert] 'operator' didn't convert. ...
 ```
 
-For non-transformed operators, the original code keeps. Please manually migrate them. [Click here](https://www.mindspore.cn/doc/note/en/master/specification_note.html#operator_api) for more information about operator mapping.
+For non-transformed operators, the original code keeps. Please manually migrate them. [Click here](https://www.mindspore.cn/doc/note/en/master/index.html#operator_api) for more information about operator mapping.
 
 
 Here is an example of the conversion report:

mindinsight/mindconverter/README_CN.md

@@ -107,7 +107,7 @@ mindconverter --in_file /home/user/model.py \
               --report /home/user/output/report
 ```
 
-转换报告中，对于未转换的代码行形式为如下，其中x, y指明的是原PyTorch脚本中代码的行、列号。对于未成功转换的算子，可参考[MindSporeAPI映射查询功能](https://www.mindspore.cn/doc/note/zh-CN/master/specification_note.html#operator_api) 手动对代码进行迁移。对于工具无法迁移的算子，会保留原脚本中的代码。
+转换报告中，对于未转换的代码行形式为如下，其中x, y指明的是原PyTorch脚本中代码的行、列号。对于未成功转换的算子，可参考[MindSporeAPI映射查询功能](https://www.mindspore.cn/doc/note/zh-CN/master/index.html#operator_api) 手动对代码进行迁移。对于工具无法迁移的算子，会保留原脚本中的代码。
 
 ```text
 line x:y: [UnConvert] 'operator' didn't convert. ...

mindinsight/ui/package.json

@@ -9,8 +9,8 @@
     "i18n:report": "vue-cli-service i18n:report --src './src/**/*.?(js|vue)' --locales './src/locales/**/*.json'"
   },
   "dependencies": {
-    "axios": "0.18.1",
-    "core-js": "3.3.2",
+    "axios": "0.19.2",
+    "core-js": "3.6.5",
     "d3": "5.9.7",
     "d3-graphviz": "3.0.4",
     "element-ui": "2.11.1",

mindinsight/ui/src/common/common-property.js

@@ -112,7 +112,7 @@ export default {
     '.edge path {stroke: rgb(120, 120, 120);}.edge polygon {fill: rgb(120, 120, 120);}' +
     '.node.aggregation > polygon {stroke: #e3aa00;fill: #ffe794;}.node.cluster.aggregation > ' +
     'rect {stroke: #e3aa00;fill: #ffe794;stroke-dasharray: 3, 3;}' +
-    '.node.cluster > rect:hover {stroke: #8df1f2;}.node > polygon {stroke: #00a5a7;fill: #rgb(141,241,242);}' +
+    '.node.cluster > rect:hover {stroke: #8df1f2;}.node > polygon {stroke: #00a5a7;fill: rgb(141,241,242);}' +
     '.node > ellipse {stroke: #4ea6e6;fill: #b8e0ff;}.node > path {stroke: #e37d29;fill: #ffd0a6;' +
     'stroke-dasharray: 3, 3;}' +
     '.hide {visibility: hidden;}.show {visibility: visible;}' +

mindinsight/ui/src/components/histogramUnit.vue

@@ -16,8 +16,12 @@ limitations under the License.
 <template>
   <div class="cl-histogram-container">
     <div class="data-show-container">
+      <div v-show="requestError"
+           class="error-msg-container">
+        {{errorMsg}}
+      </div>
       <div :id="itemId"
-           v-show="!!fullData.length"
+           v-show="!!fullData.length && !requestError"
            class="data-item"></div>
     </div>
   </div>
@@ -63,6 +67,9 @@ export default {
       zrDrawElement: {hoverDots: []},
       zr: null,
       chartTipFlag: false, // Wheather to display tips of the histogram
+      requestError: false, // Exceeded the specification
+      errorMsg: '', // Error message
+      viewResizeFlag: false, // Size reset flag
     };
   },
   computed: {},
@@ -84,7 +91,11 @@ export default {
      */
     resizeView() {
       if (this.charObj) {
-        this.charObj.resize();
+        if (this.requestError) {
+          this.viewResizeFlag = true;
+        } else {
+          this.charObj.resize();
+        }
       }
     },
     /**
@@ -139,6 +150,10 @@ export default {
       }
       this.removeTooltip();
       this.charObj.setOption(this.charOption, true);
+      if (this.viewResizeFlag) {
+        this.charObj.resize();
+        this.viewResizeFlag = false;
+      }
     },
     /**
      * Binding interaction event
@@ -825,11 +840,23 @@ export default {
      */
     updateHistogramData() {
       this.$nextTick(() => {
+        if (this.requestError) {
+          this.requestError = false;
+          this.viewResizeFlag = true;
+        }
         this.formatDataToChar();
         this.updateSampleData();
         this.sampleEventBind();
       });
     },
+    /**
+     * Show error message
+     * @param {String} errorMsg Error message
+     */
+    showRequestErrorMessage(errorMsg) {
+      this.errorMsg = errorMsg;
+      this.requestError = true;
+    },
   },
   destroyed() {
     this.clearZrData();
@@ -849,6 +876,13 @@ export default {
       width: 100%;
       height: 100%;
     }
+    .error-msg-container {
+      width: 100%;
+      height: 100%;
+      display: flex;
+      align-items: center;
+      justify-content: center;
+    }
   }
 }
 </style>

mindinsight/ui/src/locales/en-us.json

@@ -521,6 +521,7 @@
     "50545012": "The tensor data does not exist. Please refresh.",
     "50545013": "The requested data is too large. Try another dimension.",
     "50545014": "The queried tensor data has been replaced by new data. Please refresh.",
+    "50545016": "The requested tensor is too large. The total number of values exceeds 10 million and cannot be displayed.",
     "50548001": "Ascend AI Processor information query timed out.",
 
     "5054B080": "Incorrect parameter type. Please check the input parameter type.",

mindinsight/ui/src/locales/zh-cn.json

@@ -520,6 +520,7 @@
     "50545012": "张量数据不存在，请刷新。",
     "50545013": "请求的数据过大，请使用其他维度重试。",
     "50545014": "查询的张量数据已被新数据替换，请刷新。",
+    "50545016": "请求的张量过大，数值总个数超过1000万，无法显示。",
     "50548001": "昇腾AI处理器信息查询超时",
 
     "5054B080": "参数类型错误，请检查输入参数类型",

mindinsight/ui/src/services/fetcher.js

@@ -75,7 +75,7 @@ axios.interceptors.response.use(
 
         const ignoreCode = {
           ignoreError: ['50545005'],
-          regardError: ['50545013', '50545014', '5054500D'],
+          regardError: ['50545013', '50545014', '50545016', '5054500D'],
         };
 
         if (

mindinsight/ui/src/services/request-service.js

@@ -167,7 +167,7 @@ export default {
   getImageData(params) {
     return axios({
       method: 'get',
-      url: '/v1/mindinsight/datavisual/image/single-image',
+      url: 'v1/mindinsight/datavisual/image/single-image',
       params: params,
       headers: {
         ignoreError: true,

mindinsight/ui/src/views/debugger/debugger.vue

@@ -559,6 +559,7 @@ export default {
       resolve: null,
       toleranceInput: 0,
       showFilterInput: true,
+      currentNodeName: '',
     };
   },
   components: {debuggerGridTable},
@@ -806,7 +807,7 @@ export default {
         mode: 'node',
         params: {
           watch_point_id: this.curWatchPointId ? this.curWatchPointId : 0,
-          name: this.nodeName,
+          name: this.currentNodeName,
           single_node: true,
           node_type: 'leaf',
         },
@@ -820,12 +821,12 @@ export default {
               if (res.data.graph) {
                 const graph = res.data.graph;
                 if (graph.children) {
-                  this.dealTreeData(graph.children, this.nodeName);
+                  this.dealTreeData(graph.children, this.currentNodeName);
                   this.defaultCheckedArr = this.$refs.tree.getCheckedKeys();
                 }
                 this.querySingleNode(
                     JSON.parse(JSON.stringify(res.data.graph)),
-                    this.nodeName,
+                    this.currentNodeName,
                     false,
                 );
               }
@@ -1130,16 +1131,14 @@ export default {
       if (metadata.node_name !== undefined && metadata.step !== undefined) {
         const nodeName = metadata.node_name;
         if (
-          (nodeName !== this.nodeName && nodeName !== '') ||
+          (nodeName !== this.currentNodeName && nodeName !== '') ||
           this.metadata.step !== metadata.step
         ) {
           this.nodeName = nodeName ? nodeName : this.nodeName;
+          this.currentNodeName = nodeName ? nodeName : this.currentNodeName;
           this.metadata.step = metadata.step;
           if (isQuery) {
-            this.queryAllTreeData(
-              nodeName ? nodeName : this.$refs.tree.getCurrentKey(),
-              true,
-            );
+            this.queryAllTreeData(this.nodeName, true);
           }
         }
       }
@@ -1557,6 +1556,7 @@ export default {
                   }
                   this.initCondition();
                   this.nodeName = this.metadata.node_name;
+                  this.currentNodeName = this.metadata.node_name;
                   if (this.pollInit) {
                     this.pollData();
                     this.pollInit = false;

mindinsight/ui/src/views/train-manage/image.vue

@@ -163,7 +163,7 @@ export default {
       pageSizes: [8, 16, 24], // The number of records on each page is optional
       pageNum: 8, // Number of records on each page
       isReloading: false, // Manually refresh
-      imageBasePath: '/v1/mindinsight/datavisual/image/single-image?', // Relative path header of the picture
+      imageBasePath: 'v1/mindinsight/datavisual/image/single-image?', // Relative path header of the picture
     };
   },
   computed: {

mindinsight/ui/src/views/train-manage/tensor.vue

@@ -553,8 +553,8 @@ export default {
               }
             });
           },
-          () => {
-            sampleItem.showLoading = false;
+          (e) => {
+            this.freshDataErrorCallback(e, sampleItem, false);
           },
       );
     },
@@ -688,23 +688,42 @@ export default {
             });
           },
           (e) => {
-            let showLimitError = false;
-            let errorMsg = '';
-            if (
-              e.response &&
-            e.response.data &&
-            e.response.data.error_code &&
-            (e.response.data.error_code.toString() === '50545013' ||
-              e.response.data.error_code.toString() === '50545014')
-            ) {
-              showLimitError = true;
-              errorMsg = this.$t('error')[e.response.data.error_code];
-            }
-            this.clearMartixData(sampleItem, showLimitError, errorMsg);
-            sampleItem.showLoading = false;
+            this.freshDataErrorCallback(e, sampleItem, true);
           },
       );
     },
+    /**
+     * callback of fresh data
+     * @param {Object} errorData The error object
+     * @param {Object} sampleItem The object that is being operated
+     * @param {Boolean} isMartix Martix data
+     */
+    freshDataErrorCallback(errorData, sampleItem, isMartix) {
+      let showLimitError = false;
+      let errorMsg = '';
+      if (
+        errorData.response &&
+            errorData.response.data &&
+            errorData.response.data.error_code &&
+            (errorData.response.data.error_code.toString() === '50545013' ||
+              errorData.response.data.error_code.toString() === '50545014' ||
+              errorData.response.data.error_code.toString() === '50545016')
+      ) {
+        showLimitError = true;
+        errorMsg = this.$t('error')[errorData.response.data.error_code];
+      }
+      if (isMartix) {
+        this.clearMartixData(sampleItem, showLimitError, errorMsg);
+      } else {
+        this.$nextTick(() => {
+          const elementItem = this.$refs[sampleItem.ref];
+          if (elementItem) {
+            elementItem[0].showRequestErrorMessage(errorMsg);
+          }
+        });
+      }
+      sampleItem.showLoading = false;
+    },
     /**
      * Clear table display
      * @param {Object} sampleItem The object that is being operated

mindinsight/ui/src/views/train-manage/training-dashboard.vue

@@ -286,7 +286,7 @@ export default {
       originImageDataArr: [], // List of all image data.
       curImageShowSample: {}, // Image object to be displayed.
       imageRandomLoopCount: 0,
-      imageBasePath: '/v1/mindinsight/datavisual/image/single-image?', // Relative path header of the picture
+      imageBasePath: 'v1/mindinsight/datavisual/image/single-image?', // Relative path header of the picture
       autoUpdateTimer: null, // Automatic refresh timer
       histogramTag: '', // Label of the currently displayed histogram.
       allGraphData: {}, // graph Original input data
@@ -1108,13 +1108,14 @@ export default {
             e.response.data &&
             e.response.data.error_code &&
             (e.response.data.error_code.toString() === '50545013' ||
-              e.response.data.error_code.toString() === '50545014')
+              e.response.data.error_code.toString() === '50545014' ||
+              e.response.data.error_code.toString() === '50545016')
             ) {
               showLimitError = true;
-              if (e.response.data.error_code.toString() === '50545014') {
-                errorMsg = this.$t('error')[e.response.data.error_code];
-              } else {
+              if (e.response.data.error_code.toString() === '50545013') {
                 errorMsg = this.$t('tensors.tensorDashboardLimitErrorMsg');
+              } else {
+                errorMsg = this.$t('error')[e.response.data.error_code];
               }
             }
             this.$nextTick(() => {

mindinsight/utils/constant.py

@@ -82,6 +82,7 @@ class DataVisualErrors(Enum):
     TENSOR_NOT_EXIST = 18
     MAX_RESPONSE_DATA_EXCEEDED_ERROR = 19
     STEP_TENSOR_DATA_NOT_IN_CACHE = 20
+    TENSOR_TOO_LARGE = 22
 
 
 class ScriptConverterErrors(Enum):

mindinsight/utils/tensor.py

@@ -183,6 +183,8 @@ class TensorUtils:
             ParamValueError, If the length of param dims is not equal to the length of tensor dims.
             IndexError, If the param dims and tensor shape is unmatched.
         """
+        if ndarray.size == 0:
+            return ndarray
         if len(ndarray.shape) != len(dims):
             raise ParamValueError("Invalid dims. The length of param dims and tensor shape should be the same.")
         try:

mindinsight/wizard/README.md

@@ -27,7 +27,7 @@ Run the mindwizard command and answer the following questions as prompted：
 
 1. Select a network（LeNet / AlexNet / ResNet50 / ...）
 
-    1.1. Select a Loss Function（SoftmaxCrossEntropyExpand / SoftmaxCrossEntropyWithLogits / ...）
+    1.1. Select a Loss Function（SoftmaxCrossEntropyWithLogits / ...）
 
     1.2. Select a Optimizer（Adam / Momentum / SGD ...）
 
@@ -67,9 +67,8 @@ $ mindwizard project
    3: resnet50
  : 2
 >>> Please select a loss function:
-   1: SoftmaxCrossEntropyExpand
-   2: SoftmaxCrossEntropyWithLogits
- [2]: 2
+   1: SoftmaxCrossEntropyWithLogits
+ [1]: 1
 >>> Please select an optimizer:
    1: Adam
    2: Momentum

mindinsight/wizard/README_CN.md

@@ -27,7 +27,7 @@ optional arguments:
 
 1. 请选择网络（LeNet / AlexNet / ResNet50 / ...）
 
-    1.1. 请选择损失函数（SoftmaxCrossEntropyExpand / SoftmaxCrossEntropyWithLogits / ...）
+    1.1. 请选择损失函数（SoftmaxCrossEntropyWithLogits / ...）
 
     1.2. 请选择优化器（Adam / Momentum / SGD ...）
 
@@ -67,9 +67,8 @@ $ mindwizard project
    3: resnet50
  : 2
 >>> Please select a loss function:
-   1: SoftmaxCrossEntropyExpand
-   2: SoftmaxCrossEntropyWithLogits
- [2]: 2
+   1: SoftmaxCrossEntropyWithLogits
+ [1]: 1
 >>> Please select an optimizer:
    1: Adam
    2: Momentum

mindinsight/wizard/network/alexnet.py

@@ -14,5 +14,5 @@ class Network(GenericNetwork):
     """Network code generator."""
     name = 'alexnet'
     supported_datasets = ['Cifar10', 'ImageNet']
-    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits', 'SoftmaxCrossEntropyExpand']
+    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits']
     supported_optimizers = ['Momentum', 'Adam', 'SGD']

mindinsight/wizard/network/lenet.py

@@ -20,5 +20,5 @@ class Network(GenericNetwork):
     """Network code generator."""
     name = 'lenet'
     supported_datasets = ['MNIST']
-    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits', 'SoftmaxCrossEntropyExpand']
+    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits']
     supported_optimizers = ['Momentum', 'Adam', 'SGD']

mindinsight/wizard/network/resnet50.py

@@ -14,5 +14,5 @@ class Network(GenericNetwork):
     """Network code generator."""
     name = 'resnet50'
     supported_datasets = ['Cifar10', 'ImageNet']
-    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits', 'SoftmaxCrossEntropyExpand']
+    supported_loss_functions = ['SoftmaxCrossEntropyWithLogits']
     supported_optimizers = ['Momentum', 'Adam', 'SGD']

requirements.txt

@@ -1,4 +1,3 @@
-attrdict>=2.0.1
 Click>=7.0
 Flask>=1.1.1
 Flask-Cors>=3.0.8
@@ -13,8 +12,7 @@ protobuf>=3.8.0
 psutil>=5.6.1
 six>=1.12.0
 Werkzeug>=1.0.0
-tabulate>=0.8.6
 pandas>=1.0.4
 yapf>=0.30.0
 treelib>=1.6.1
-grpcio>=1.29.0
+grpcio>=1.27.3

tests/st/func/lineagemgr/cache/__init__.py

@@ -1,14 +0,0 @@
-# Copyright 2020 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================

tests/st/func/lineagemgr/cache/test_lineage_cache.py

@@ -1,100 +0,0 @@
-# Copyright 2020 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-"""
-Function:
-    Test the query module about lineage information.
-Usage:
-    The query module test should be run after lineagemgr/collection/model/test_model_lineage.py
-    pytest lineagemgr
-"""
-from unittest import TestCase
-
-import pytest
-
-from mindinsight.datavisual.data_transform.data_manager import DataManager
-from mindinsight.lineagemgr.cache_item_updater import LineageCacheItemUpdater
-from mindinsight.lineagemgr.model import filter_summary_lineage, get_summary_lineage
-
-from ..test_model import LINEAGE_INFO_RUN1, LINEAGE_FILTRATION_EXCEPT_RUN, \
-    LINEAGE_FILTRATION_RUN1, LINEAGE_FILTRATION_RUN2
-from ..conftest import BASE_SUMMARY_DIR
-from .....ut.lineagemgr.querier import event_data
-from .....utils.tools import assert_equal_lineages
-
-
-@pytest.mark.usefixtures("create_summary_dir")
-class TestModelApi(TestCase):
-    """Test get lineage from data_manager."""
-    @classmethod
-    def setup_class(cls):
-        data_manager = DataManager(BASE_SUMMARY_DIR)
-        data_manager.register_brief_cache_item_updater(LineageCacheItemUpdater())
-        data_manager.start_load_data().join()
-
-        cls._data_manger = data_manager
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_get_summary_lineage(self):
-        """Test the interface of get_summary_lineage."""
-        total_res = get_summary_lineage(data_manager=self._data_manger, summary_dir="./run1")
-        expect_total_res = LINEAGE_INFO_RUN1
-        assert_equal_lineages(expect_total_res, total_res, self.assertDictEqual)
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_filter_summary_lineage(self):
-        """Test the interface of filter_summary_lineage."""
-        expect_result = {
-            'customized': event_data.CUSTOMIZED__1,
-            'object': [
-                LINEAGE_FILTRATION_EXCEPT_RUN,
-                LINEAGE_FILTRATION_RUN1,
-                LINEAGE_FILTRATION_RUN2
-            ],
-            'count': 3
-        }
-
-        search_condition = {
-            'sorted_name': 'summary_dir'
-        }
-        res = filter_summary_lineage(data_manager=self._data_manger, search_condition=search_condition)
-        expect_objects = expect_result.get('object')
-        for idx, res_object in enumerate(res.get('object')):
-            expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
-        assert_equal_lineages(expect_result, res, self.assertDictEqual)
-
-        expect_result = {
-            'customized': {},
-            'object': [],
-            'count': 0
-        }
-
-        search_condition = {
-            'summary_dir': {
-                "in": ['./dir_with_empty_lineage']
-            }
-        }
-        res = filter_summary_lineage(data_manager=self._data_manger, search_condition=search_condition)
-        assert_equal_lineages(expect_result, res, self.assertDictEqual)

tests/st/func/lineagemgr/collection/model/test_model_lineage.py

@@ -27,7 +27,7 @@ from unittest import TestCase, mock
 
 import numpy as np
 import pytest
-from mindinsight.lineagemgr.model import get_summary_lineage
+from mindinsight.lineagemgr.model import filter_summary_lineage
 from mindinsight.lineagemgr.common.exceptions.error_code import LineageErrors
 from mindinsight.utils.exceptions import MindInsightException
 
@@ -37,10 +37,11 @@ from mindspore.dataset.engine import MindDataset
 from mindspore.nn import Momentum, SoftmaxCrossEntropyWithLogits, WithLossCell
 from mindspore.train.callback import ModelCheckpoint, RunContext, SummaryStep, _ListCallback
 from mindspore.train.summary import SummaryRecord
-from tests.utils.lineage_writer.model_lineage import AnalyzeObject, EvalLineage, TrainLineage
 
-from ...conftest import SUMMARY_DIR, SUMMARY_DIR_2, SUMMARY_DIR_3
 from .train_one_step import TrainOneStep
+from ...conftest import SUMMARY_DIR, SUMMARY_DIR_2, SUMMARY_DIR_3, BASE_SUMMARY_DIR, LINEAGE_DATA_MANAGER
+from ......utils.lineage_writer.model_lineage import AnalyzeObject, EvalLineage, TrainLineage
+from ......utils.tools import get_relative_path
 
 
 @pytest.mark.usefixtures("create_summary_dir")
@@ -76,6 +77,13 @@ class TestModelLineage(TestCase):
             "version": "v1"
         }
 
+        train_id = get_relative_path(SUMMARY_DIR, BASE_SUMMARY_DIR)
+        cls._search_condition = {
+            'summary_dir': {
+                'eq': train_id
+            }
+        }
+
     @pytest.mark.scene_train(2)
     @pytest.mark.level0
     @pytest.mark.platform_arm_ascend_training
@@ -90,13 +98,17 @@ class TestModelLineage(TestCase):
         train_callback = TrainLineage(SUMMARY_DIR, True, self.user_defined_info)
         train_callback.initial_learning_rate = 0.12
         train_callback.end(RunContext(self.run_context))
-        res = get_summary_lineage(summary_dir=SUMMARY_DIR)
-        assert res.get('hyper_parameters', {}).get('epoch') == 10
+
+        LINEAGE_DATA_MANAGER.start_load_data().join()
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, self._search_condition)
+        assert res.get('object')[0].get('model_lineage', {}).get('epoch') == 10
         run_context = self.run_context
         run_context['epoch_num'] = 14
         train_callback.end(RunContext(run_context))
-        res = get_summary_lineage(summary_dir=SUMMARY_DIR)
-        assert res.get('hyper_parameters', {}).get('epoch') == 14
+
+        LINEAGE_DATA_MANAGER.start_load_data().join()
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, self._search_condition)
+        assert res.get('object')[0].get('model_lineage', {}).get('epoch') == 14
 
     @pytest.mark.scene_eval(3)
     @pytest.mark.level0
@@ -186,12 +198,13 @@ class TestModelLineage(TestCase):
         run_context_customized['train_network'] = net
         train_callback.begin(RunContext(run_context_customized))
         train_callback.end(RunContext(run_context_customized))
-        res = get_summary_lineage(summary_dir=SUMMARY_DIR)
-        assert res.get('hyper_parameters', {}).get('loss_function') \
-               == 'SoftmaxCrossEntropyWithLogits'
-        assert res.get('algorithm', {}).get('network') == 'ResNet'
-        assert res.get('hyper_parameters', {}).get('optimizer') == 'Momentum'
 
+        LINEAGE_DATA_MANAGER.start_load_data().join()
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, self._search_condition)
+        assert res.get('object')[0].get('model_lineage', {}).get('loss_function') \
+               == 'SoftmaxCrossEntropyWithLogits'
+        assert res.get('object')[0].get('model_lineage', {}).get('network') == 'ResNet'
+        assert res.get('object')[0].get('model_lineage', {}).get('optimizer') == 'Momentum'
 
     @pytest.mark.scene_exception(1)
     @pytest.mark.level0

tests/st/func/lineagemgr/conftest.py

@@ -21,6 +21,9 @@ import tempfile
 
 import pytest
 
+from mindinsight.datavisual.data_transform.data_manager import DataManager
+from mindinsight.lineagemgr.cache_item_updater import LineageCacheItemUpdater
+
 from ....utils import mindspore
 from ....utils.mindspore.dataset.engine.serializer_deserializer import SERIALIZED_PIPELINE
 
@@ -31,6 +34,9 @@ SUMMARY_DIR = os.path.join(BASE_SUMMARY_DIR, 'run1')
 SUMMARY_DIR_2 = os.path.join(BASE_SUMMARY_DIR, 'run2')
 SUMMARY_DIR_3 = os.path.join(BASE_SUMMARY_DIR, 'except_run')
 
+LINEAGE_DATA_MANAGER = DataManager(BASE_SUMMARY_DIR)
+LINEAGE_DATA_MANAGER.register_brief_cache_item_updater(LineageCacheItemUpdater())
+
 COLLECTION_MODULE = 'TestModelLineage'
 API_MODULE = 'TestModelApi'
 DATASET_GRAPH = SERIALIZED_PIPELINE

tests/st/func/lineagemgr/test_model.py

@@ -24,20 +24,18 @@ import os
 from unittest import TestCase
 import pytest
 
-from mindinsight.lineagemgr.model import filter_summary_lineage, get_summary_lineage
-from mindinsight.lineagemgr.common.exceptions.exceptions import (LineageFileNotFoundError, LineageParamSummaryPathError,
-                                                                 LineageParamTypeError, LineageParamValueError,
-                                                                 LineageSearchConditionParamError)
+from mindinsight.lineagemgr.model import filter_summary_lineage
 from mindinsight.datavisual.data_transform import data_manager
 from mindinsight.lineagemgr.cache_item_updater import LineageCacheItemUpdater
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageSearchConditionParamError
 from mindinsight.lineagemgr.model import get_flattened_lineage
 
-from .conftest import BASE_SUMMARY_DIR, DATASET_GRAPH, SUMMARY_DIR, SUMMARY_DIR_2
+from .conftest import BASE_SUMMARY_DIR, DATASET_GRAPH, SUMMARY_DIR, SUMMARY_DIR_2, LINEAGE_DATA_MANAGER
 from ....ut.lineagemgr.querier import event_data
-from ....utils.tools import assert_equal_lineages
+from ....utils.tools import assert_equal_lineages, get_relative_path
 
 LINEAGE_INFO_RUN1 = {
-    'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'run1'),
+    'summary_dir': './run1',
     'metric': {
         'accuracy': 0.78
     },
@@ -67,7 +65,7 @@ LINEAGE_INFO_RUN1 = {
     'dataset_graph': DATASET_GRAPH
 }
 LINEAGE_FILTRATION_EXCEPT_RUN = {
-    'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'except_run'),
+    'summary_dir': './except_run',
     'model_lineage': {
         'loss_function': 'SoftmaxCrossEntropyWithLogits',
         'train_dataset_path': None,
@@ -86,10 +84,11 @@ LINEAGE_FILTRATION_EXCEPT_RUN = {
         'metric': {},
         'dataset_mark': 2
     },
-    'dataset_graph': DATASET_GRAPH
+    'dataset_graph': DATASET_GRAPH,
+    'added_info': {}
 }
 LINEAGE_FILTRATION_RUN1 = {
-    'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'run1'),
+    'summary_dir': './run1',
     'model_lineage': {
         'loss_function': 'SoftmaxCrossEntropyWithLogits',
         'train_dataset_path': None,
@@ -114,10 +113,11 @@ LINEAGE_FILTRATION_RUN1 = {
         },
         'dataset_mark': 2
     },
-    'dataset_graph': DATASET_GRAPH
+    'dataset_graph': DATASET_GRAPH,
+    'added_info': {}
 }
 LINEAGE_FILTRATION_RUN2 = {
-    'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'run2'),
+    'summary_dir': './run2',
     'model_lineage': {
         'loss_function': "SoftmaxCrossEntropyWithLogits",
         'train_dataset_path': None,
@@ -138,7 +138,8 @@ LINEAGE_FILTRATION_RUN2 = {
         },
         'dataset_mark': 3
     },
-    'dataset_graph': DATASET_GRAPH
+    'dataset_graph': DATASET_GRAPH,
+    'added_info': {}
 }
 
 
@@ -149,12 +150,12 @@ class TestModelApi(TestCase):
     @classmethod
     def setup_class(cls):
         """The initial process."""
-        cls.dir_with_empty_lineage = os.path.join(
+        cls._dir_with_empty_lineage = os.path.join(
             BASE_SUMMARY_DIR, 'dir_with_empty_lineage'
         )
-        os.makedirs(cls.dir_with_empty_lineage)
+        os.makedirs(cls._dir_with_empty_lineage)
         ms_file = os.path.join(
-            cls.dir_with_empty_lineage, 'empty.summary.1581499502.bms_MS'
+            cls._dir_with_empty_lineage, 'empty.summary.1581499502.bms_MS'
         )
         lineage_file = ms_file + '_lineage'
         with open(ms_file, mode='w'):
@@ -162,156 +163,11 @@ class TestModelApi(TestCase):
         with open(lineage_file, mode='w'):
             pass
 
-        cls.empty_dir = os.path.join(BASE_SUMMARY_DIR, 'empty_dir')
-        os.makedirs(cls.empty_dir)
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_get_summary_lineage(self):
-        """Test the interface of get_summary_lineage."""
-        total_res = get_summary_lineage(None, SUMMARY_DIR)
-        partial_res1 = get_summary_lineage(None, SUMMARY_DIR, ['hyper_parameters'])
-        partial_res2 = get_summary_lineage(None, SUMMARY_DIR, ['metric', 'algorithm'])
-        expect_total_res = LINEAGE_INFO_RUN1
-        expect_partial_res1 = {
-            'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'run1'),
-            'hyper_parameters': {
-                'optimizer': 'Momentum',
-                'learning_rate': 0.12,
-                'loss_function': 'SoftmaxCrossEntropyWithLogits',
-                'epoch': 14,
-                'parallel_mode': 'stand_alone',
-                'device_num': 2,
-                'batch_size': 32
-            }
-        }
-        expect_partial_res2 = {
-            'summary_dir': os.path.join(BASE_SUMMARY_DIR, 'run1'),
-            'metric': {
-                'accuracy': 0.78
-            },
-            'algorithm': {
-                'network': 'ResNet'
-            }
-        }
-        assert_equal_lineages(expect_total_res, total_res, self.assertDictEqual)
-        assert_equal_lineages(expect_partial_res1, partial_res1, self.assertDictEqual)
-        assert_equal_lineages(expect_partial_res2, partial_res2, self.assertDictEqual)
-
-        # the lineage summary file is empty
-        result = get_summary_lineage(None, self.dir_with_empty_lineage)
-        assert {} == result
-
-        # keys is empty list
-        expect_result = {
-            'summary_dir': SUMMARY_DIR
-        }
-        result = get_summary_lineage(None, SUMMARY_DIR, [])
-        assert expect_result == result
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_get_summary_lineage_exception_1(self):
-        """Test the interface of get_summary_lineage with exception."""
-        # summary path does not exist
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path does not exist or is not a dir.',
-            get_summary_lineage,
-            None,
-            '/tmp/fake/dir'
-        )
-
-        # summary path is relative path
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            get_summary_lineage,
-            None,
-            'tmp'
-        )
-
-        # the type of input param is invalid
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            get_summary_lineage,
-            None,
-            ['/root/linage1', '/root/lineage2']
-        )
-
-        # summary path is empty str
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            get_summary_lineage,
-            None,
-            '',
-            keys=None
-        )
+        empty_dir = os.path.join(BASE_SUMMARY_DIR, 'empty_dir')
+        os.makedirs(empty_dir)
+        cls._empty_train_id = get_relative_path(empty_dir, LINEAGE_DATA_MANAGER.summary_base_dir)
 
-        # summary path invalid
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            get_summary_lineage,
-            None,
-            '\\',
-            keys=None
-        )
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_get_summary_lineage_exception_2(self):
-        """Test the interface of get_summary_lineage with exception."""
-        # keys is invalid
-        self.assertRaisesRegex(
-            LineageParamValueError,
-            'Keys must be in',
-            get_summary_lineage,
-            None,
-            SUMMARY_DIR,
-            ['metric', 'fake_name']
-        )
-
-        self.assertRaisesRegex(
-            LineageParamTypeError,
-            'Element of keys must be str.',
-            get_summary_lineage,
-            None,
-            SUMMARY_DIR,
-            [1, 2, 3]
-        )
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_get_summary_lineage_exception_3(self):
-        """Test the interface of get_summary_lineage with exception."""
-        for keys in [0, 0.1, True, (3, 4), {'a': 'b'}]:
-            self.assertRaisesRegex(
-                LineageParamTypeError,
-                'Keys must be list.',
-                get_summary_lineage,
-                None,
-                SUMMARY_DIR,
-                keys
-            )
+        LINEAGE_DATA_MANAGER.start_load_data().join()
 
     @pytest.mark.level0
     @pytest.mark.platform_arm_ascend_training
@@ -334,18 +190,7 @@ class TestModelApi(TestCase):
         search_condition = {
             'sorted_name': 'summary_dir'
         }
-        res = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition)
-        expect_objects = expect_result.get('object')
-        for idx, res_object in enumerate(res.get('object')):
-            expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
-        assert_equal_lineages(expect_result, res, self.assertDictEqual)
-
-        expect_result = {
-            'customized': {},
-            'object': [],
-            'count': 0
-        }
-        res = filter_summary_lineage(None, self.dir_with_empty_lineage)
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition)
         expect_objects = expect_result.get('object')
         for idx, res_object in enumerate(res.get('object')):
             expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
@@ -362,8 +207,8 @@ class TestModelApi(TestCase):
         search_condition = {
             'summary_dir': {
                 'in': [
-                    SUMMARY_DIR,
-                    SUMMARY_DIR_2
+                    get_relative_path(SUMMARY_DIR, LINEAGE_DATA_MANAGER.summary_base_dir),
+                    get_relative_path(SUMMARY_DIR_2, LINEAGE_DATA_MANAGER.summary_base_dir)
                 ]
             },
             'metric/accuracy': {
@@ -383,7 +228,7 @@ class TestModelApi(TestCase):
             ],
             'count': 2
         }
-        partial_res = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition)
+        partial_res = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition)
         expect_objects = expect_result.get('object')
         for idx, res_object in enumerate(partial_res.get('object')):
             expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
@@ -421,7 +266,7 @@ class TestModelApi(TestCase):
             ],
             'count': 2
         }
-        partial_res = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition)
+        partial_res = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition)
         expect_objects = expect_result.get('object')
         for idx, res_object in enumerate(partial_res.get('object')):
             expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
@@ -450,7 +295,7 @@ class TestModelApi(TestCase):
             ],
             'count': 3
         }
-        partial_res1 = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition1)
+        partial_res1 = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition1)
         expect_objects = expect_result.get('object')
         for idx, res_object in enumerate(partial_res1.get('object')):
             expect_objects[idx]['model_lineage']['dataset_mark'] = res_object['model_lineage'].get('dataset_mark')
@@ -469,9 +314,30 @@ class TestModelApi(TestCase):
             'object': [],
             'count': 0
         }
-        partial_res2 = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition2)
+        partial_res2 = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition2)
         assert expect_result == partial_res2
 
+    @pytest.mark.level0
+    @pytest.mark.platform_arm_ascend_training
+    @pytest.mark.platform_x86_gpu_training
+    @pytest.mark.platform_x86_ascend_training
+    @pytest.mark.platform_x86_cpu
+    @pytest.mark.env_single
+    def test_filter_summary_lineage_condition_4(self):
+        """Test the abnormal execution of the filter_summary_lineage interface."""
+        expect_result = {
+            'customized': {},
+            'object': [],
+            'count': 0
+        }
+        search_condition = {
+            'summary_dir': {
+                'eq': self._empty_train_id
+            }
+        }
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition)
+        assert expect_result == res
+
     @pytest.mark.level0
     @pytest.mark.platform_arm_ascend_training
     @pytest.mark.platform_x86_gpu_training
@@ -480,10 +346,10 @@ class TestModelApi(TestCase):
     @pytest.mark.env_single
     def test_filter_summary_lineage_with_lineage_type(self):
         """Test the interface of filter_summary_lineage with lineage_type."""
-        summary_dir = os.path.join(BASE_SUMMARY_DIR, 'except_run')
+        train_id = './except_run'
         search_condition = {
             'summary_dir': {
-                'in': [summary_dir]
+                'in': [train_id]
             },
             'lineage_type': {
                 'eq': 'dataset'
@@ -493,50 +359,16 @@ class TestModelApi(TestCase):
             'customized': {},
             'object': [
                 {
-                    'summary_dir': summary_dir,
-                    'dataset_graph': DATASET_GRAPH
+                    'summary_dir': train_id,
+                    'dataset_graph': DATASET_GRAPH,
+                    'added_info': {}
                 }
             ],
             'count': 1
         }
-        res = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition)
+        res = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition)
         assert expect_result == res
 
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_filter_summary_lineage_exception_1(self):
-        """Test the abnormal execution of the filter_summary_lineage interface."""
-        # summary base dir is relative path
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            filter_summary_lineage,
-            None,
-            'relative_path'
-        )
-
-        # summary base dir does not exist
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path does not exist or is not a dir.',
-            filter_summary_lineage,
-            None,
-            '/path/does/not/exist'
-        )
-
-        # no summary log file under summary_base_dir
-        self.assertRaisesRegex(
-            LineageFileNotFoundError,
-            'There is no summary log file under summary_base_dir.',
-            filter_summary_lineage,
-            None,
-            self.empty_dir
-        )
-
     @pytest.mark.level0
     @pytest.mark.platform_arm_ascend_training
     @pytest.mark.platform_x86_gpu_training
@@ -553,8 +385,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The search_condition element summary_dir should be dict.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -566,8 +397,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The sorted_name have to exist when sorted_type exists.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -577,8 +407,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'Invalid search_condition type, it should be dict.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -590,8 +419,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The limit must be int.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -611,8 +439,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The offset must be int.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -626,8 +453,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The search attribute not supported.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -648,8 +474,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The sorted_type must be ascending or descending',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -663,8 +488,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The compare condition should be in',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -678,8 +502,7 @@ class TestModelApi(TestCase):
             LineageSearchConditionParamError,
             'The parameter metric/accuracy is invalid.',
             filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
+            LINEAGE_DATA_MANAGER,
             search_condition
         )
 
@@ -690,31 +513,6 @@ class TestModelApi(TestCase):
     @pytest.mark.platform_x86_cpu
     @pytest.mark.env_single
     def test_filter_summary_lineage_exception_5(self):
-        """Test the abnormal execution of the filter_summary_lineage interface."""
-        # the summary dir is invalid in search condition
-        search_condition = {
-            'summary_dir': {
-                'in': [
-                    'xxx'
-                ]
-            }
-        }
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            filter_summary_lineage,
-            None,
-            BASE_SUMMARY_DIR,
-            search_condition
-        )
-
-    @pytest.mark.level0
-    @pytest.mark.platform_arm_ascend_training
-    @pytest.mark.platform_x86_gpu_training
-    @pytest.mark.platform_x86_ascend_training
-    @pytest.mark.platform_x86_cpu
-    @pytest.mark.env_single
-    def test_filter_summary_lineage_exception_6(self):
         """Test the abnormal execution of the filter_summary_lineage interface."""
         # gt > lt
         search_condition1 = {
@@ -728,7 +526,7 @@ class TestModelApi(TestCase):
             'object': [],
             'count': 0
         }
-        partial_res1 = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition1)
+        partial_res1 = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition1)
         assert expect_result == partial_res1
 
         # the (offset + 1) * limit > count
@@ -744,7 +542,7 @@ class TestModelApi(TestCase):
             'object': [],
             'count': 2
         }
-        partial_res2 = filter_summary_lineage(None, BASE_SUMMARY_DIR, search_condition2)
+        partial_res2 = filter_summary_lineage(LINEAGE_DATA_MANAGER, search_condition2)
         assert expect_result == partial_res2
 
     @pytest.mark.level0
@@ -753,7 +551,7 @@ class TestModelApi(TestCase):
     @pytest.mark.platform_x86_ascend_training
     @pytest.mark.platform_x86_cpu
     @pytest.mark.env_single
-    def test_filter_summary_lineage_exception_7(self):
+    def test_filter_summary_lineage_exception_6(self):
         """Test the abnormal execution of the filter_summary_lineage interface."""
         condition_keys = ["summary_dir", "lineage_type", "loss_function", "optimizer", "network", "dataset_mark"]
         for condition_key in condition_keys:
@@ -767,8 +565,7 @@ class TestModelApi(TestCase):
                 LineageSearchConditionParamError,
                 f'The parameter {condition_key} is invalid. Its operation should be `eq`, `in` or `not_in`.',
                 filter_summary_lineage,
-                None,
-                BASE_SUMMARY_DIR,
+                LINEAGE_DATA_MANAGER,
                 search_condition
             )
 
@@ -778,7 +575,7 @@ class TestModelApi(TestCase):
     @pytest.mark.platform_x86_ascend_training
     @pytest.mark.platform_x86_cpu
     @pytest.mark.env_single
-    def test_filter_summary_lineage_exception_8(self):
+    def test_filter_summary_lineage_exception_7(self):
         """Test the abnormal execution of the filter_summary_lineage interface."""
         invalid_lineage_types = ['xxx', None]
         for lineage_type in invalid_lineage_types:
@@ -791,8 +588,7 @@ class TestModelApi(TestCase):
                 LineageSearchConditionParamError,
                 "The parameter lineage_type is invalid. It should be 'dataset' or 'model'.",
                 filter_summary_lineage,
-                None,
-                BASE_SUMMARY_DIR,
+                LINEAGE_DATA_MANAGER,
                 search_condition
             )
 
@@ -802,7 +598,7 @@ class TestModelApi(TestCase):
     @pytest.mark.platform_x86_ascend_training
     @pytest.mark.platform_x86_cpu
     @pytest.mark.env_single
-    def test_filter_summary_lineage_exception_9(self):
+    def test_filter_summary_lineage_exception_8(self):
         """Test the abnormal execution of the filter_summary_lineage interface."""
         invalid_sorted_names = ['xxx', 'metric_', 1]
         for sorted_name in invalid_sorted_names:
@@ -813,8 +609,7 @@ class TestModelApi(TestCase):
                 LineageSearchConditionParamError,
                 'The sorted_name must be in',
                 filter_summary_lineage,
-                None,
-                BASE_SUMMARY_DIR,
+                LINEAGE_DATA_MANAGER,
                 search_condition
             )
 

tests/st/func/wizard/test_alexnet.py

@@ -18,6 +18,7 @@ Function:
     Test the various combinations based on AlexNet.
 """
 import os
+import shutil
 import pytest
 
 from mindinsight.wizard.base.utility import load_network_maker
@@ -49,21 +50,6 @@ class TestAlexNet:
                    'optimizer': 'SGD',
                    'dataset': 'Cifar10'},
         'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Momentum',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Adam',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'SGD',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
     }, {
         'config': {'loss': 'SoftmaxCrossEntropyWithLogits',
                    'optimizer': 'Momentum',
@@ -79,21 +65,6 @@ class TestAlexNet:
                    'optimizer': 'SGD',
                    'dataset': 'ImageNet'},
         'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Momentum',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Adam',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'SGD',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
     }])
     def test_combinations(self, params):
         """Do testing."""
@@ -105,11 +76,18 @@ class TestAlexNet:
         network_maker = load_network_maker(network_maker_name)
         network_maker.configure(config)
 
-        self.source_files = network_maker.generate(**config)
+        source_files = network_maker.generate(**config)
+
+        self.output_dir = os.path.realpath('test_folder')
+        for source_file in source_files:
+            source_file.write(self.output_dir)
 
-        self.check_scripts()
-        self.check_src(dataset_loader_name, config)
-        self.check_train_eval_readme(config['dataset'], config['loss'], config['optimizer'])
+        try:
+            self.check_scripts()
+            self.check_src(dataset_loader_name, config)
+            self.check_train_eval_readme(config['dataset'], config['loss'], config['optimizer'])
+        finally:
+            shutil.rmtree(self.output_dir)
 
     def check_src(self, dataset_name, config):
         """Check src file."""
@@ -118,19 +96,27 @@ class TestAlexNet:
         config_optimizer_is_right = False
         network_is_right = False
         generator_lr_is_right = False
-        for source_file in self.source_files:
-            if source_file.file_relative_path == 'src/dataset.py':
-                if dataset_name in source_file.content:
-                    dataset_is_right = True
-            if source_file.file_relative_path == os.path.join('src', NETWORK_NAME.lower()+'.py'):
-                network_is_right = True
-            if source_file.file_relative_path == 'src/generator_lr.py':
-                generator_lr_is_right = True
-            if source_file.file_relative_path == 'src/config.py':
-                content = source_file.content
-
-                config_dataset_is_right = self._check_config_dataset(config, content)
-                config_optimizer_is_right = self._check_config_optimizer(config, content)
+
+        sub_output_dir_list = os.walk(self.output_dir)
+        for sub_output_dir in sub_output_dir_list:
+            for sub_output_file in sub_output_dir[-1]:
+                content_dir = os.path.relpath(
+                    os.path.join(sub_output_dir[0], sub_output_file),
+                    self.output_dir)
+
+                if content_dir == 'src/dataset.py':
+                    with open(os.path.realpath(os.path.join(self.output_dir, content_dir))) as file:
+                        if dataset_name in file.read():
+                            dataset_is_right = True
+                if content_dir == os.path.join('src', NETWORK_NAME.lower() + '.py'):
+                    network_is_right = True
+                if content_dir == 'src/generator_lr.py':
+                    generator_lr_is_right = True
+                if content_dir == 'src/config.py':
+                    with open(os.path.realpath(os.path.join(self.output_dir, content_dir))) as file:
+                        content = file.read()
+                        config_dataset_is_right = self._check_config_dataset(config, content)
+                        config_optimizer_is_right = self._check_config_optimizer(config, content)
 
         assert dataset_is_right
         assert config_dataset_is_right
@@ -171,21 +157,32 @@ class TestAlexNet:
         train_is_right = False
         eval_is_right = False
         readme_is_right = False
-        for source_file in self.source_files:
-            if source_file.file_relative_path == 'train.py':
-                content = source_file.content
-                if 'alexnet' in content and loss_name in content and optimizer_name in content:
-                    train_is_right = True
-
-            if source_file.file_relative_path == 'eval.py':
-                content = source_file.content
-                if 'alexnet' in content and loss_name in content:
-                    eval_is_right = True
-
-            if source_file.file_relative_path == 'README.md':
-                content = source_file.content
-                if 'AlexNet' in content and dataset_name in content:
-                    readme_is_right = True
+
+        sub_output_dir_list = os.walk(self.output_dir)
+        for sub_output_dir in sub_output_dir_list:
+            for sub_output_file in sub_output_dir[-1]:
+                content_dir = os.path.relpath(
+                    os.path.join(sub_output_dir[0], sub_output_file),
+                    self.output_dir)
+
+                if content_dir == 'train.py':
+                    with open(os.path.realpath(os.path.join(self.output_dir, content_dir))) as file:
+                        content = file.read()
+                        if 'alexnet' in content and loss_name in content and optimizer_name in content:
+                            train_is_right = True
+
+                if content_dir == 'eval.py':
+                    with open(os.path.realpath(os.path.join(self.output_dir, content_dir))) as file:
+                        content = file.read()
+                        if 'alexnet' in content and loss_name in content:
+                            eval_is_right = True
+
+                if content_dir == 'README.md':
+                    with open(os.path.realpath(os.path.join(self.output_dir, content_dir))) as file:
+                        content = file.read()
+                        if 'AlexNet' in content and dataset_name in content:
+                            readme_is_right = True
+
         assert train_is_right
         assert eval_is_right
         assert readme_is_right
@@ -200,19 +197,25 @@ class TestAlexNet:
         exist_run_standalone_train = False
         exist_run_standalone_train_gpu = False
 
-        for source_file in self.source_files:
-            if source_file.file_relative_path == 'scripts/run_distribute_train.sh':
-                exist_run_distribute_train = True
-            if source_file.file_relative_path == 'scripts/run_distribute_train_gpu.sh':
-                exist_run_distribute_train_gpu = True
-            if source_file.file_relative_path == 'scripts/run_eval.sh':
-                exist_run_eval = True
-            if source_file.file_relative_path == 'scripts/run_eval_gpu.sh':
-                exist_run_eval_gpu = True
-            if source_file.file_relative_path == 'scripts/run_standalone_train.sh':
-                exist_run_standalone_train = True
-            if source_file.file_relative_path == 'scripts/run_standalone_train_gpu.sh':
-                exist_run_standalone_train_gpu = True
+        sub_output_dir_list = os.walk(self.output_dir)
+        for sub_output_dir in sub_output_dir_list:
+            for sub_output_file in sub_output_dir[-1]:
+                content_dir = os.path.relpath(
+                    os.path.join(sub_output_dir[0], sub_output_file),
+                    self.output_dir)
+
+                if content_dir == 'scripts/run_distribute_train.sh':
+                    exist_run_distribute_train = True
+                if content_dir == 'scripts/run_distribute_train_gpu.sh':
+                    exist_run_distribute_train_gpu = True
+                if content_dir == 'scripts/run_eval.sh':
+                    exist_run_eval = True
+                if content_dir == 'scripts/run_eval_gpu.sh':
+                    exist_run_eval_gpu = True
+                if content_dir == 'scripts/run_standalone_train.sh':
+                    exist_run_standalone_train = True
+                if content_dir == 'scripts/run_standalone_train_gpu.sh':
+                    exist_run_standalone_train_gpu = True
 
         assert exist_run_distribute_train
         assert exist_run_distribute_train_gpu

tests/st/func/wizard/test_lenet.py

@@ -48,21 +48,6 @@ class TestLeNet:
                    'optimizer': 'SGD',
                    'dataset': 'MNIST'},
         'dataset_loader_name': 'MnistDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Momentum',
-                   'dataset': 'MNIST'},
-        'dataset_loader_name': 'MnistDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Adam',
-                   'dataset': 'MNIST'},
-        'dataset_loader_name': 'MnistDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyWithLogits',
-                   'optimizer': 'SGD',
-                   'dataset': 'MNIST'},
-        'dataset_loader_name': 'MnistDataset'
     }])
     def test_combinations(self, params):
         """Do testing."""

tests/st/func/wizard/test_resnet50.py

@@ -49,21 +49,6 @@ class TestResNet50:
                    'optimizer': 'SGD',
                    'dataset': 'Cifar10'},
         'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Momentum',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Adam',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'SGD',
-                   'dataset': 'Cifar10'},
-        'dataset_loader_name': 'Cifar10Dataset'
     }, {
         'config': {'loss': 'SoftmaxCrossEntropyWithLogits',
                    'optimizer': 'Momentum',
@@ -79,21 +64,6 @@ class TestResNet50:
                    'optimizer': 'SGD',
                    'dataset': 'ImageNet'},
         'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Momentum',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'Adam',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
-    }, {
-        'config': {'loss': 'SoftmaxCrossEntropyExpand',
-                   'optimizer': 'SGD',
-                   'dataset': 'ImageNet'},
-        'dataset_loader_name': 'ImageFolderDataset'
     }])
     def test_combinations(self, params):
         """Do testing."""

tests/ut/backend/lineagemgr/test_lineage_api.py

@@ -14,7 +14,6 @@
 # ============================================================================
 """Test the module of lineage_api."""
 import json
-import os
 from unittest import TestCase, mock
 
 from flask import Response
@@ -73,21 +72,21 @@ class TestSearchModel(TestCase):
     @mock.patch('mindinsight.backend.lineagemgr.lineage_api.filter_summary_lineage')
     def test_search_model_success(self, *args):
         """Test the success of model_success."""
-        base_dir = '/path/to/test_lineage_summary_dir_base'
         args[0].return_value = {
             'object': [
                 {
-                    'summary_dir': base_dir,
-                    'model_lineage': LINEAGE_FILTRATION_BASE
+                    'summary_dir': './',
+                    'model_lineage': LINEAGE_FILTRATION_BASE,
+                    'added_info': {}
                 },
                 {
-                    'summary_dir': os.path.join(base_dir, 'run1'),
-                    'model_lineage': LINEAGE_FILTRATION_RUN1
+                    'summary_dir': './run1',
+                    'model_lineage': LINEAGE_FILTRATION_RUN1,
+                    'added_info': {}
                 }
             ],
             'count': 2
         }
-        args[1].SUMMARY_BASE_DIR = base_dir
 
         body_data = {
             'limit': 10,
@@ -101,11 +100,13 @@ class TestSearchModel(TestCase):
             'object': [
                 {
                     'summary_dir': './',
-                    'model_lineage': LINEAGE_FILTRATION_BASE
+                    'model_lineage': LINEAGE_FILTRATION_BASE,
+                    'added_info': {}
                 },
                 {
                     'summary_dir': './run1',
-                    'model_lineage': LINEAGE_FILTRATION_RUN1
+                    'model_lineage': LINEAGE_FILTRATION_RUN1,
+                    'added_info': {}
                 }
             ],
             'count': 2

tests/ut/lineagemgr/querier/test_querier.py

@@ -13,6 +13,7 @@
 # limitations under the License.
 # ============================================================================
 """Test the querier module."""
+import os
 import time
 
 from unittest import TestCase, mock
@@ -22,7 +23,7 @@ from google.protobuf.json_format import ParseDict
 
 import mindinsight.datavisual.proto_files.mindinsight_lineage_pb2 as summary_pb2
 from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamTypeError, LineageQuerierParamException
-from mindinsight.lineagemgr.lineage_parser import LineageOrganizer
+from mindinsight.lineagemgr.lineage_parser import LineageParser
 from mindinsight.lineagemgr.querier.querier import Querier
 from mindinsight.lineagemgr.summary.lineage_summary_analyzer import LineageInfo
 
@@ -104,6 +105,7 @@ def create_filtration_result(summary_dir, train_event_dict,
             "user_defined": {}
         },
         "dataset_graph": dataset_dict,
+        'added_info': {}
     }
     return filtration_result
 
@@ -162,42 +164,42 @@ LINEAGE_INFO_1 = {
     'dataset_graph': event_data.DATASET_DICT_0
 }
 LINEAGE_FILTRATION_0 = create_filtration_result(
-    '/path/to/summary0',
+    './summary0',
     event_data.EVENT_TRAIN_DICT_0,
     event_data.EVENT_EVAL_DICT_0,
     event_data.METRIC_0,
     event_data.DATASET_DICT_0
 )
 LINEAGE_FILTRATION_1 = create_filtration_result(
-    '/path/to/summary1',
+    './summary1',
     event_data.EVENT_TRAIN_DICT_1,
     event_data.EVENT_EVAL_DICT_1,
     event_data.METRIC_1,
     event_data.DATASET_DICT_0
 )
 LINEAGE_FILTRATION_2 = create_filtration_result(
-    '/path/to/summary2',
+    './summary2',
     event_data.EVENT_TRAIN_DICT_2,
     event_data.EVENT_EVAL_DICT_2,
     event_data.METRIC_2,
     event_data.DATASET_DICT_0
 )
 LINEAGE_FILTRATION_3 = create_filtration_result(
-    '/path/to/summary3',
+    './summary3',
     event_data.EVENT_TRAIN_DICT_3,
     event_data.EVENT_EVAL_DICT_3,
     event_data.METRIC_3,
     event_data.DATASET_DICT_0
 )
 LINEAGE_FILTRATION_4 = create_filtration_result(
-    '/path/to/summary4',
+    './summary4',
     event_data.EVENT_TRAIN_DICT_4,
     event_data.EVENT_EVAL_DICT_4,
     event_data.METRIC_4,
     event_data.DATASET_DICT_0
 )
 LINEAGE_FILTRATION_5 = {
-    "summary_dir": '/path/to/summary5',
+    "summary_dir": './summary5',
     "model_lineage": {
         "loss_function":
             event_data.EVENT_TRAIN_DICT_5['train_lineage']['hyper_parameters']['loss_function'],
@@ -219,10 +221,11 @@ LINEAGE_FILTRATION_5 = {
         "dataset_mark": '2',
         "user_defined": {}
     },
-    "dataset_graph": event_data.DATASET_DICT_0
+    "dataset_graph": event_data.DATASET_DICT_0,
+    "added_info": {}
 }
 LINEAGE_FILTRATION_6 = {
-    "summary_dir": '/path/to/summary6',
+    "summary_dir": './summary6',
     "model_lineage": {
         "loss_function": None,
         "train_dataset_path": None,
@@ -243,15 +246,16 @@ LINEAGE_FILTRATION_6 = {
         "dataset_mark": '2',
         "user_defined": {}
     },
-    "dataset_graph": event_data.DATASET_DICT_0
+    "dataset_graph": event_data.DATASET_DICT_0,
+    "added_info": {}
 }
 
 
 class TestQuerier(TestCase):
     """Test the class of `Querier`."""
+    _MOCK_DATA_MANAGER = MagicMock()
 
     @mock.patch('mindinsight.lineagemgr.lineage_parser.SummaryPathParser.get_lineage_summaries')
-    @mock.patch('mindinsight.lineagemgr.lineage_parser.SummaryWatcher.list_summary_directories')
     @mock.patch('mindinsight.lineagemgr.lineage_parser.LineageSummaryAnalyzer.get_user_defined_info')
     @mock.patch('mindinsight.lineagemgr.lineage_parser.LineageSummaryAnalyzer.get_summary_infos')
     @mock.patch('mindinsight.lineagemgr.lineage_parser.FileHandler')
@@ -263,139 +267,23 @@ class TestQuerier(TestCase):
             event_data.EVENT_DATASET_DICT_0
         )
         args[1].return_value = []
-        args[3].return_value = ['path']
+        args[2].return_value = ['path']
         mock_file_handler = MagicMock()
         mock_file_handler.size = 1
 
-        args[2].return_value = [{'relative_path': './', 'update_time': 1}]
-        single_summary_path = '/path/to/summary0'
-        lineage_objects = LineageOrganizer(summary_base_dir=single_summary_path).super_lineage_objs
-        self.single_querier = Querier(lineage_objects)
+        summary_dir = '/path/test/'
 
         lineage_infos = get_lineage_infos()
         args[0].side_effect = lineage_infos
-        summary_base_dir = '/path/to'
-        relative_dirs = []
+        lineage_objects = {}
         for i in range(7):
-            relative_dirs.append(dict(relative_path=f'./summary{i}', update_time=time.time() - i))
-        args[2].return_value = relative_dirs
-        lineage_objects = LineageOrganizer(summary_base_dir=summary_base_dir).super_lineage_objs
-        self.multi_querier = Querier(lineage_objects)
-
-    def test_get_summary_lineage_success_1(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [LINEAGE_INFO_0]
-        result = self.single_querier.get_summary_lineage()
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_success_2(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [LINEAGE_INFO_0]
-        result = self.single_querier.get_summary_lineage()
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_success_3(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [
-            {
-                'summary_dir': '/path/to/summary0',
-                'model': event_data.EVENT_TRAIN_DICT_0['train_lineage']['model'],
-                'algorithm': event_data.EVENT_TRAIN_DICT_0['train_lineage']['algorithm']
-            }
-        ]
-        result = self.single_querier.get_summary_lineage(
-            filter_keys=['model', 'algorithm']
-        )
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_success_4(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [
-            LINEAGE_INFO_0,
-            LINEAGE_INFO_1,
-            {
-                'summary_dir': '/path/to/summary2',
-                **event_data.EVENT_TRAIN_DICT_2['train_lineage'],
-                'metric': event_data.METRIC_2,
-                'valid_dataset': event_data.EVENT_EVAL_DICT_2['evaluation_lineage']['valid_dataset'],
-                'dataset_graph': event_data.DATASET_DICT_0
-            },
-            {
-                'summary_dir': '/path/to/summary3',
-                **event_data.EVENT_TRAIN_DICT_3['train_lineage'],
-                'metric': event_data.METRIC_3,
-                'valid_dataset': event_data.EVENT_EVAL_DICT_3['evaluation_lineage']['valid_dataset'],
-                'dataset_graph': event_data.DATASET_DICT_0
-            },
-            {
-                'summary_dir': '/path/to/summary4',
-                **event_data.EVENT_TRAIN_DICT_4['train_lineage'],
-                'metric': event_data.METRIC_4,
-                'valid_dataset': event_data.EVENT_EVAL_DICT_4['evaluation_lineage']['valid_dataset'],
-                'dataset_graph': event_data.DATASET_DICT_0
-            },
-            {
-                'summary_dir': '/path/to/summary5',
-                **event_data.EVENT_TRAIN_DICT_5['train_lineage'],
-                'metric': {},
-                'valid_dataset': {},
-                'dataset_graph': event_data.DATASET_DICT_0
-            },
-            {
-                'summary_dir': '/path/to/summary6',
-                'hyper_parameters': {},
-                'algorithm': {},
-                'model': {},
-                'train_dataset': {},
-                'metric': event_data.METRIC_5,
-                'valid_dataset': event_data.EVENT_EVAL_DICT_5['evaluation_lineage']['valid_dataset'],
-                'dataset_graph': event_data.DATASET_DICT_0
-            }
-        ]
-        result = self.multi_querier.get_summary_lineage()
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_success_5(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [LINEAGE_INFO_1]
-        result = self.multi_querier.get_summary_lineage(
-            summary_dir='/path/to/summary1'
-        )
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_success_6(self):
-        """Test the success of get_summary_lineage."""
-        expected_result = [
-            {
-                'summary_dir': '/path/to/summary0',
-                'hyper_parameters': event_data.EVENT_TRAIN_DICT_0['train_lineage']['hyper_parameters'],
-                'train_dataset': event_data.EVENT_TRAIN_DICT_0['train_lineage']['train_dataset'],
-                'metric': event_data.METRIC_0,
-                'valid_dataset': event_data.EVENT_EVAL_DICT_0['evaluation_lineage']['valid_dataset']
-            }
-        ]
-        filter_keys = [
-            'metric', 'hyper_parameters', 'train_dataset', 'valid_dataset'
-        ]
-        result = self.multi_querier.get_summary_lineage(
-            summary_dir='/path/to/summary0', filter_keys=filter_keys
-        )
-        assert_equal_lineages(expected_result, result, self.assertListEqual)
-
-    def test_get_summary_lineage_fail(self):
-        """Test the function of get_summary_lineage with exception."""
-        filter_keys = ['xxx']
-        self.assertRaises(
-            LineageQuerierParamException,
-            self.multi_querier.get_summary_lineage,
-            filter_keys=filter_keys
-        )
+            train_id = f'./summary{i}'
+            summary_dir = os.path.join(summary_dir, train_id)
+            update_time = time.time() - i
+            lineage_parser = LineageParser(train_id, summary_dir, update_time)
+            lineage_objects.update({train_id: lineage_parser.super_lineage_obj})
 
-        self.assertRaises(
-            LineageQuerierParamException,
-            self.multi_querier.get_summary_lineage,
-            summary_dir='xxx'
-        )
+        self.multi_querier = Querier(lineage_objects)
 
     def test_filter_summary_lineage_success_1(self):
         """Test the success of filter_summary_lineage."""

tests/ut/lineagemgr/querier/test_query_model.py

@@ -133,6 +133,7 @@ class TestLineageObj(TestCase):
         )
         expected_result['model_lineage']['dataset_mark'] = None
         expected_result.pop('dataset_graph')
+        expected_result.pop('added_info')
         result = self.lineage_obj.to_model_lineage_dict()
         assert_equal_lineages(expected_result, result, self.assertDictEqual)
 

tests/ut/lineagemgr/test_model.py

@@ -16,224 +16,60 @@
 from unittest import TestCase, mock
 from unittest.mock import MagicMock
 
-from mindinsight.lineagemgr.model import get_summary_lineage, filter_summary_lineage, \
-    _convert_relative_path_to_abspath, get_flattened_lineage
-from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamSummaryPathError, \
-    LineageFileNotFoundError, LineageSummaryParseException, LineageQuerierParamException, \
-    LineageQuerySummaryDataError, LineageSearchConditionParamError, LineageParamTypeError, \
-    LineageParamValueError
+from mindinsight.lineagemgr.model import filter_summary_lineage, get_flattened_lineage
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageSummaryParseException, \
+    LineageQuerierParamException, LineageQuerySummaryDataError, LineageSearchConditionParamError, LineageParamTypeError
 from mindinsight.lineagemgr.common.path_parser import SummaryPathParser
 from ...st.func.lineagemgr.test_model import LINEAGE_FILTRATION_RUN1, LINEAGE_FILTRATION_RUN2
 
 
-class TestModel(TestCase):
-    """Test the function of get_summary_lineage and filter_summary_lineage."""
-
-    @mock.patch('mindinsight.lineagemgr.model.Querier')
-    @mock.patch('mindinsight.lineagemgr.model.LineageParser')
-    @mock.patch('os.path.isdir')
-    def test_get_summary_lineage_success(self, isdir_mock, parser_mock, qurier_mock):
-        """Test the function of get_summary_lineage."""
-        isdir_mock.return_value = True
-        parser_mock.return_value = MagicMock()
-
-        mock_querier = MagicMock()
-        qurier_mock.return_value = mock_querier
-        mock_querier.get_summary_lineage.return_value = [{'algorithm': {'network': 'ResNet'}}]
-        summary_dir = '/path/to/summary_dir'
-        result = get_summary_lineage(None, summary_dir, keys=['algorithm'])
-        self.assertEqual(result, {'algorithm': {'network': 'ResNet'}})
-
-    def test_get_summary_lineage_failed(self):
-        """Test get_summary_lineage failed."""
-        invalid_path = '../fake_dir'
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            get_summary_lineage,
-            None,
-            invalid_path
-        )
-
-    @mock.patch('mindinsight.lineagemgr.common.utils.validate_path')
-    @mock.patch.object(SummaryPathParser, 'get_lineage_summaries')
-    def test_get_summary_lineage_failed2(self, mock_summary, mock_valid):
-        """Test get_summary_lineage failed."""
-        mock_summary.return_value = []
-        mock_valid.return_value = '/path/to/summary/dir'
-        self.assertRaisesRegex(
-            LineageFileNotFoundError,
-            'no summary log file under summary_dir',
-            get_summary_lineage,
-            None,
-            '/path/to/summary_dir'
-        )
-
-    @mock.patch('mindinsight.lineagemgr.lineage_parser.FileHandler')
-    @mock.patch('mindinsight.lineagemgr.lineage_parser.LineageParser._parse_summary_log')
-    @mock.patch('mindinsight.lineagemgr.common.utils.validate_path')
-    @mock.patch.object(SummaryPathParser, 'get_lineage_summaries')
-    def test_get_summary_lineage_failed3(self,
-                                         mock_summary,
-                                         mock_valid,
-                                         mock_parser,
-                                         mock_file_handler):
-        """Test get_summary_lineage failed."""
-        mock_summary.return_value = ['/path/to/summary/file']
-        mock_valid.return_value = '/path/to/summary_dir'
-        mock_parser.return_value = None
-        mock_file_handler = MagicMock()
-        mock_file_handler.size = 1
-        result = get_summary_lineage(None, '/path/to/summary_dir')
-        assert {} == result
-
-    @mock.patch('mindinsight.lineagemgr.model.validate_path')
-    def test_convert_relative_path_to_abspath(self, validate_path_mock):
-        """Test the function of converting realtive path to abspath."""
-        validate_path_mock.return_value = '/path/to/summary_base_dir/summary_dir'
-        summary_base_dir = '/path/to/summary_base_dir'
-        search_condition = {
-            'summary_dir': {
-                'in': ['/path/to/summary_base_dir']
-            }
-        }
-        result = _convert_relative_path_to_abspath(summary_base_dir,
-                                                   search_condition)
-        self.assertDictEqual(
-            result, {'summary_dir': {'in': ['/path/to/summary_base_dir/summary_dir']}})
-
-        search_condition = {
-            'summary_dir': {
-                'in': ['./summary_dir']
-            }
-        }
-        result = _convert_relative_path_to_abspath(summary_base_dir, search_condition)
-        self.assertDictEqual(
-            result, {'summary_dir': {'in': ['/path/to/summary_base_dir/summary_dir']}}
-        )
-
-        search_condition = {
-            'summary_dir': {
-                'eq': '/summary_dir'
-            }
-        }
-        result = _convert_relative_path_to_abspath(summary_base_dir, search_condition)
-        self.assertDictEqual(
-            result, {'summary_dir': {'eq': '/path/to/summary_base_dir/summary_dir'}})
-
-        search_condition = {
-            'summary_dir': None
-        }
-        result = _convert_relative_path_to_abspath(summary_base_dir, search_condition)
-        self.assertDictEqual(
-            result, search_condition
-        )
-
-
 class TestFilterAPI(TestCase):
     """Test the function of filter_summary_lineage."""
-    @mock.patch('mindinsight.lineagemgr.model.LineageOrganizer')
+    _MOCK_DATA_MANAGER = MagicMock()
+
     @mock.patch('mindinsight.lineagemgr.model.Querier')
     @mock.patch('mindinsight.lineagemgr.lineage_parser.SummaryPathParser.get_lineage_summaries')
-    @mock.patch('mindinsight.lineagemgr.model._convert_relative_path_to_abspath')
-    @mock.patch('mindinsight.lineagemgr.model.normalize_summary_dir')
-    def test_filter_summary_lineage(self, validate_path_mock, convert_path_mock,
-                                    latest_summary_mock, qurier_mock, organizer_mock):
+    def test_filter_summary_lineage(self, latest_summary_mock, qurier_mock):
         """Test the function of filter_summary_lineage."""
-        convert_path_mock.return_value = {
-            'summary_dir': {
-                'in': ['/path/to/summary_base_dir']
-            },
-            'loss': {
-                'gt': 2.0
-            }
-        }
-        organizer_mock = MagicMock()
-        organizer_mock.super_lineage_objs = None
-        validate_path_mock.return_value = True
         latest_summary_mock.return_value = ['/path/to/summary_base_dir/summary_dir']
         mock_querier = MagicMock()
         qurier_mock.return_value = mock_querier
         mock_querier.filter_summary_lineage.return_value = [{'loss': 3.0}]
 
-        summary_base_dir = '/path/to/summary_base_dir'
-        result = filter_summary_lineage(None, summary_base_dir)
+        result = filter_summary_lineage(self._MOCK_DATA_MANAGER)
         self.assertEqual(result, [{'loss': 3.0}])
 
-    def test_invalid_path(self):
-        """Test filter_summary_lineage with invalid path."""
-        invalid_path = '../fake_dir'
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'The summary path is invalid.',
-            filter_summary_lineage,
-            None,
-            invalid_path
-        )
-
     @mock.patch('mindinsight.lineagemgr.model.validate_condition')
-    @mock.patch('mindinsight.lineagemgr.model.normalize_summary_dir')
-    def test_invalid_search_condition(self, mock_path, mock_valid):
+    def test_invalid_search_condition(self, mock_valid):
         """Test filter_summary_lineage with invalid invalid param."""
-        mock_path.return_value = None
         mock_valid.side_effect = LineageParamTypeError(
             'Invalid search_condition type.')
         self.assertRaisesRegex(
             LineageSearchConditionParamError,
             'Invalid search_condition type.',
             filter_summary_lineage,
-            None,
-            '/path/to/summary/dir',
+            self._MOCK_DATA_MANAGER,
             'invalid_condition'
         )
 
-    @mock.patch('mindinsight.lineagemgr.model.validate_search_model_condition')
-    @mock.patch('mindinsight.lineagemgr.model.validate_condition')
-    @mock.patch('mindinsight.lineagemgr.common.utils.validate_path')
-    @mock.patch('mindinsight.lineagemgr.model._convert_relative_path_to_abspath')
-    def test_failed_to_convert_path(self, mock_convert, *args):
+    def test_failed_to_get_summary_files(self):
         """Test filter_summary_lineage with invalid invalid param."""
-        mock_convert.side_effect = LineageParamValueError('invalid path')
-        args[0].return_value = None
-        self.assertRaisesRegex(
-            LineageParamSummaryPathError,
-            'invalid path',
-            filter_summary_lineage,
-            None,
-            '/path/to/summary/dir',
-            {}
-        )
-
-    @mock.patch('mindinsight.lineagemgr.model._convert_relative_path_to_abspath')
-    @mock.patch('mindinsight.lineagemgr.model.validate_search_model_condition')
-    @mock.patch('mindinsight.lineagemgr.model.validate_condition')
-    @mock.patch('mindinsight.lineagemgr.model.normalize_summary_dir')
-    @mock.patch.object(SummaryPathParser, 'get_lineage_summaries')
-    def test_failed_to_get_summary_filesh(self, mock_parse, *args):
-        """Test filter_summary_lineage with invalid invalid param."""
-        path = '/path/to/summary/dir'
-        mock_parse.return_value = []
-        args[0].return_value = path
-        self.assertRaisesRegex(
-            LineageFileNotFoundError,
-            'There is no summary log file under summary_base_dir.',
-            filter_summary_lineage,
-            None,
-            path
-        )
+        default_result = {
+            'customized': {},
+            'object': [],
+            'count': 0
+        }
+        assert default_result == filter_summary_lineage(self._MOCK_DATA_MANAGER)
 
-    @mock.patch('mindinsight.lineagemgr.model._convert_relative_path_to_abspath')
     @mock.patch('mindinsight.lineagemgr.model.validate_search_model_condition')
     @mock.patch('mindinsight.lineagemgr.model.validate_condition')
-    @mock.patch('mindinsight.lineagemgr.model.normalize_summary_dir')
     @mock.patch.object(SummaryPathParser, 'get_lineage_summaries')
     @mock.patch('mindinsight.lineagemgr.model.Querier')
-    def test_failed_to_querier(self, mock_query, mock_parse, *args):
+    def test_failed_to_querier(self, mock_query, *args):
         """Test filter_summary_lineage with invalid invalid param."""
         mock_query.side_effect = LineageSummaryParseException()
-        mock_parse.return_value = ['/path/to/summary/file']
         args[0].return_value = None
-        res = filter_summary_lineage(None, '/path/to/summary')
+        res = filter_summary_lineage(self._MOCK_DATA_MANAGER)
         assert res == {'object': [], 'count': 0}
 
         mock_query.side_effect = LineageQuerierParamException(['keys'], 'key')
@@ -241,8 +77,7 @@ class TestFilterAPI(TestCase):
             LineageQuerySummaryDataError,
             'Filter summary lineage failed.',
             filter_summary_lineage,
-            None,
-            '/path/to/summary/dir'
+            self._MOCK_DATA_MANAGER
         )
 
     @mock.patch('mindinsight.lineagemgr.model.filter_summary_lineage')
@@ -251,8 +86,7 @@ class TestFilterAPI(TestCase):
         mock_data = {
             'object': [LINEAGE_FILTRATION_RUN1, LINEAGE_FILTRATION_RUN2]
         }
-        mock_datamanager = MagicMock()
-        mock_datamanager.summary_base_dir = '/tmp/'
+        mock_data_manager = MagicMock()
         mock_filter_summary_lineage.return_value = mock_data
-        result = get_flattened_lineage(mock_datamanager, None)
+        result = get_flattened_lineage(mock_data_manager)
         assert result.get('[U]info') == ['info1', None]

tests/utils/tools.py

@@ -19,12 +19,14 @@ import io
 import os
 import shutil
 import json
+from pathlib import Path
 
 from urllib.parse import urlencode
-
 import numpy as np
 from PIL import Image
 
+from mindinsight.lineagemgr.common.exceptions.exceptions import LineageParamValueError
+
 
 def get_url(url, params):
     """
@@ -138,3 +140,25 @@ def assert_equal_lineages(lineages1, lineages2, assert_func, decimal_num=2):
     else:
         deal_float_for_dict(lineages1, lineages2, decimal_num)
     assert_func(lineages1, lineages2)
+
+
+def get_relative_path(path, base_path):
+    """
+    Get relative path based on base_path.
+
+    Args:
+        path (str): absolute path.
+        base_path: absolute base path.
+
+    Returns:
+        str, relative path based on base_path.
+
+    """
+    try:
+        r_path = str(Path(path).relative_to(Path(base_path)))
+    except ValueError:
+        raise LineageParamValueError("The path %r does not start with %r." % (path, base_path))
+
+    if r_path == ".":
+        r_path = ""
+    return os.path.join("./", r_path)