1. New a thread to load detail info. Loading detail info takes too much time,
so the summary list and lineage can not be loaded timely.
2. Add a status for DetailCacheManager to indicate it is INIT, LOADING or DONE.
3. Update UT/ST.
For max processes used for computing, we need to make sure every summary directory has a process to load data. We also should not use too many processes to avoid system problems (eg. out of memory). So we calc the max processes cnt in _calc_default_max_processes_cnt
For _load_data_in_thread_wrapper, because self._load_data_in_thread() will create process pool when loading files, we can not afford to run multiple self._load_data_in_thread() simultaneously. So we use a lock to make sure that only one self._load_data_in_thread() is running.
Main features:
1. Use the ComputingResourceManager to manage all computing workers.
2. Ensure fair worker number between summary directories at first. So every summary directories in cache will be loaded simultaneously.
3. When a summary directory is loaded, it's worker will be released, and other unfinished summary directories can use the released workers to speed up. This way we solved the slow worker problem.
Code changes:
1. Added computing_resource_mgr.py
2. Passed ComputingResourceManager instances instead of workers_count
3. Simplified the _load_single_fine() function a bit.
1. To accelerate summary file parsing, multiple processes are used. As the first step to mindinsight parsing performance optimization, we only made changes to _load_single_file function.
2. This PR will imporve summary parsing throughput dramatically (about cpu_count times)
3. Changes are mainly about _load_single_file function
In the future, a more global concurrent computing framework is needed for mindinsight. See the gitee wiki doc for details.