01bb751425
[to #43653669 ]feat: auto build docker images
auto build docker images
aone 任务： https://test.aone.alibaba-inc.com/jobs/1824567?buildId=143470479
修改任务分支
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9566518
* [to #43653669 ]feat: auto build docker images
3 years ago
35548bd492
[to #43875101 ]
msdataset add coco dataset
unify taskdataset and ms dataset
fix hf datasets
3 years ago
f9c1e5e296
[to #43875101 ] fix datasets error: unexpected keyworkd namespace
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9665865
3 years ago
9d0b38b4e4
[to #42322933 ] lazy load on trainer
3 years ago
21fa71baf0
[to #42322933 ] add/refactor nlp models source code and finetune
1. add sbert,veco,palm,space source code
2. support sbert sequence classification, token classification finetune
3. support veco sequence classification finetune
4. support palm nlg finetune
evaluation result: https://sheet.alibaba-inc.com/#/sheet/f7fdcc7f22bd5105 sheet:Maas
5. add ut for finetunes
6. add veco's taskdataset processor
7. add a common trainer for nlp, and a specific trainer for veco
8. merge some duplicate codes of models, preprocessors, pipelines
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9574105
* add basic class of hook&metrics
* pre-commit passed
* change some comments
* pre commit passed
* 1. remove accuracy's groups 2. remove useless hooks 3. simplify priorities
* pre-commit passed
* fix a comment
* Merge branch 'master' into finetune_hooks_metrics
# Conflicts:
# modelscope/metainfo.py
* pre-commit passed
* add basic class of hook&metrics
* pre-commit passed
* change some comments
* pre commit passed
* 1. remove accuracy's groups 2. remove useless hooks 3. simplify priorities
* pre-commit passed
* fix a comment
* Merge branch 'feat/finetune' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into feat/finetune
* mv hooks related to modelscope/trainers/hooks
* mv priority back
* add torch mdoel base and test
* update hooks, trainer, import_util
* add torch epoch based trainer and dis utils
* add hooks
* fix warmup
* format code stype and fix warmup and add warmup unittest
* fix impls
* pre-commit check passed
* update hook and add EpochBasedTrainer
* add trainer unittest
* Merge branch 'feat/add_hooks' into feat/add_task
# Conflicts:
# modelscope/models/base_torch.py
# modelscope/trainers/hooks/hook.py
# modelscope/trainers/trainer.py
* update unittest name
* rewrite taskdataset to trainer
* fix trainer and add unittest
* add unittest
* code: run to forward
* run through... but ugly code
* arrange some cls
* fix some errs
* revert some mistakes
* init check in
* Merge branch 'feat/add_hooks' into feat/add_task
# Conflicts:
# modelscope/trainers/trainer.py
* test with bigger epoch and size
* add the default metrics class
* move build metrics code to a method
* merge add_task
* merge origin add_task
* add device initialization
* remove preprocessor arg for bool
* add task models
* move metric collect logic to metrics class
* pre-commit passed
* fix cr comments
* precommit passed
* add task models
* Merge remote-tracking branch 'origin/feat/add_task' into feat/backbone_head
* add comment
* change comment formats.
* fix comments
* fix ut bug
* fix comments
* add wrapper check
* fix comments
* pre commit passed
* fix cr comments
* solve a loop import problem
* fix ut bug
* fix ut errors
* change dummydataset to msdataset
* precommit passed
* merge add task
* backbone-head is build, model is not correctly loaded
* model load states matched
* result matched
* lint
* add veco/palm_v2 code
* merge master
* merge master success running
* add repr model name level
* Merge branch 'feat/veco_palm' into feat/finetune_sbert_veco
* model test for training
* add token-classification metric add formal ut
* fix running bug
* finetune and pipeline are working with backbone-head
* add nli
* add missing code
* finetune and pipeline are working with backbone-head
* Merge branch 'feat/backbone_head' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/backbone_head
* add a test repo for pr
* remove merge conflicted file
* remove merge conflicted file 1
* lint check
* import error
* none type bug fix
* forward input unpacking or dict bug
* move head into models, add build_backbone with registry, no base method
* merge master
* feat: 1. add interleave dataset method 2. support multiple dataset in trainer.build_dataset 3. support 3 sub tasks in sequence_classification task
* unfinished
* update the task model structure in NLP field
* merge master
* update by comments
* keep the default model id as current on production
* unfinished
* unfinished
* veco can run
* Merge remote-tracking branch 'origin/master' into feat/backbone_head
* add taskmodel for module management
* remove forward_input_is_dict
* unfinished
* token classification started
* update base model structure
* move space to backbone
* remove 'type' in build_from_cfg method
* test update
* bug fix
* on tesing, mess code
* Merge branch 'feat/backbone_head' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metrics/builder.py
# modelscope/models/__init__.py
# modelscope/models/nlp/__init__.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
# requirements/multi-modal.txt
* add missing merge
* add sofa source code
* refactor
* add veco task dataset
* add veco task dataset
* pre-commit passed
* fix bug of log
* add some features
* merge master
* bug fix
* refine nlp models
* fix the training error
* unfinished
* refactor pipeline
* Merge branch 'feat/backbone_head' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metrics/builder.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/structbert/modeling_sbert.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/preprocessors/base.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
* Merge commit 'ab04ceafc5453ce7daa9aa09e37a55f703072a10' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metainfo.py
# modelscope/metrics/builder.py
# modelscope/models/__init__.py
# modelscope/models/base/base_torch_model.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/space/model/intent_unified_transformer.py
# modelscope/models/nlp/backbones/space/model/model_base.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sequence_classification.py
# modelscope/models/nlp/space/__init__.py
# modelscope/models/nlp/space_for_dialog_intent_prediction.py
# modelscope/models/nlp/space_for_dialog_modeling.py
# modelscope/models/nlp/space_for_dialog_state_tracking.py
# modelscope/models/nlp/task_model.py
# modelscope/pipelines/nlp/sentiment_classification_pipeline.py
# modelscope/preprocessors/base.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
* revert changes
* unify sentnece classification postprocess
* revert some changes, move some model files
* pipeline first case run through
* ws pipeline passed
* Merge branch 'feat/refactor_nlp_730' into feat/finetune_sbert_veco
* finetune
* revert code
* revert some code
* ws finetune started, only the accuracy is weird
* Merge branch 'feat/veco_taskdataset' into feat/finetune_sbert_veco
# Conflicts:
# modelscope/task_datasets/veco_dataset.py
# tests/taskdataset/test_veco_dataset.py
* veco+nli finetune started
* Merge branch 'master' into feat/finetune_sbert_veco
# Conflicts:
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sbert_for_token_classification.py
# modelscope/models/nlp/sbert_for_zero_shot_classification.py
# modelscope/models/nlp/space/space_for_dialog_intent_prediction.py
# modelscope/models/nlp/space/space_for_dialog_modeling.py
# modelscope/trainers/trainer.py
* add trainer for nlp
* trainer: dataset params passed into preprocessor
* test passed by nlptrainer
* fix some bugs
* fix some bugs
* add backbone/head subclass
* fix regression bugs
* fix bug in token-cls finetune
* support cfg modification
* fix bug
* fix bug
* update requirements
* add some comments and fix some t
* add some comments and revert a argument
* split to two test files
* revert code
* fixbug in precessor
(cherry picked from commit 7a648d096e
3 years ago
6e9dcc6327
[to #42322933 ]numpy version
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9598631
3 years ago
e93339ea87
[to #42322933 ]Merge request from 仲理:feat/product_feature
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9515599
3 years ago
743e876981
[to #43660556 ] msdataset数据集加载
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9552632
* load csv dataset from modelscoop
3 years ago
590d531484
[to #43115513 ] requirements refine and preparation for v0.3 release
* remove tensorflow numpy from audio requirements
* add audio requirements to all
* auto set model to eval model for pipeline
* add audio requirement check hint for easyasr and kwsbp
* fix docs build error
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9561021
3 years ago
d55525bfb6
[to #43112771 ] requirements check and lazy import support
3 years ago
fc90bf0d1a
[to #43554786 ]fix: test error is not detected in gate test, protobuf version to (3, 3.21.0) for tensorflow
限制protobuf版本，修复单元测试有error返回值为0问题
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9510263
* fix test error is not detected in gate test, protobuf version to (3, 3.21.0)
3 years ago
e62cd756df
[to #42322933 ] relax requirements
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9407594
3 years ago
8e51a073a6
[to #42966122 ] requirements enchanment and self-host repo support
* add self-hosted repo:
* add extra requirements for different field and reduce necessary requirements
* update docker file with so required by audio
* add requirements checker which will be used later when implement lazy import
* remove repeated requirements and replace opencv-python-headless with opencv-python
example usage:
```shell
pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383
3 years ago
76c6ff6329
[to #42675838 ]merge model hub code
合并model hub 代码
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9096493
3 years ago
99fb503695
[to #42322933 ] Add Palm2.0 model.
接入支持中英文的 Palm2.0 模型，复用 text-generation-pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9066550
3 years ago
31498c1d6a
[to #41669377 ] add speech AEC pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8973072
* [to #41669377 ] docs and tools refinement and release
1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307
* [to #41669377 ] add pipeline tutorial and fix bugs
1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301
* refine doc
* feat: add audio aec pipeline and preprocessor
* feat: add audio aec model classes
* feat: add audio aec loss functions
* refactor:delete no longer used loss function
* [to #42281043 ] support kwargs in pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8949062
* support kwargs in pipeline
* update develop doc with CR instruction
* Merge branch 'release/0.1' into dev/aec
* style: reformat code by pre-commit tools
* feat:support maas_lib pipeline auto downloading model
* test:add aec test case as sample code
* feat:aec pipeline use config from maashub
* feat:aec pipeline use feature parameters from maashub
* update setup.cfg to disable PEP8 rule W503 in flake8 and yapf
* format:fix double quoted strings, indent issues and optimize import
* refactor:extract some constant in aec pipeline
* refactor: delete no longer used __main__ statement
* chore:change all Chinese comments to English
* fix: change file name style to lower case
* refactor: rename model name
* feat:load C++ .so from LD_LIBRARY_PATH
* feat:register PROPROCESSOR for LinearAECAndFbank
* refactory:move aec process from postprocess() to forward() and update comments
* refactory:add more readable error message when audio sample rate is not 16000
* fix: package maas_lib renamed to modelscope in import statement
* feat: optimize the error message of audio layer classes
* format: delete empty lines
* refactor: rename audio preprocessor and optimize error message
* refactor: change aec model id to damo/speech_dfsmn_aec_psm_16k
* refactor: change sample audio file url to public oss
* Merge branch 'master' into dev/aec
* feat: add output info for aec pipeline
* fix: normalize output audio data to [-1.0, 1.0]
* refactor:use constant from ModelFile
* feat: AEC pipeline can use c++ lib in current working directory and the test will download it
* fix: c++ downloading should work wherever test is triggerd
3 years ago
b4fc38e1b9
[to #42461396 ] add Pillow version constraint and update hub version
1. use Pillow >= 6.2.0
2. change skip test msg for image caption
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9041194
3 years ago
1f6b376599
[to #42373878 ] refactor maaslib to modelscope
1. refactor maaslib to modelscope
2. fix UT error
3. support pipeline which does not register default model
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8988388
3 years ago
235880f300
[to #42339763 ] merge pydataset into maas-lib
* merge pydataset to the repo
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8955999
3 years ago
5995cc4607
add PyDataset support
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8868644
3 years ago
c4bfd6cced
[to #41999503 ] refine doc and requirements for linux and mac
1. refine quick start and pipeline doc
2. remove tf pytorch easynlp from requirements
3. lazy import for torch and tensorflow
4. test successfully on linux and mac intel cpu
5. update api doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8882373
3 years ago
25a2028b54
[to #41401401 ] modelhub and Trainer support
* add trainer interface
* add trainer script
* add model init support for pipelineadd pipeline tutorial and fix bugs
* add text classification evaluation to maas lib
* add quickstart and prepare env doc
* relax requirements for torch and sentencepiece
* merge release/0.1 and fix conflict
* modelhub support for model and pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8868339
3 years ago
5e469008fd
[to #41401401 ] add preprocessor, model and pipeline
* add preprocessor module
* add model base and builder
* update task constant
* add load image preprocessor and its dependency
* add pipeline interface and UT covered
* support default pipeline for task
* add image matting pipeline
* refine nlp tokenize interface
* add nlp pipeline
* fix UT failed
* add test for Compose
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8769235
* add preprocessor module
* add test for Compose
* fix citest error
* fix abs class error
* add model base and builder
* update task constant
* add load image preprocessor and its dependency
* add pipeline interface and UT covered
* support default pipeline for task
* refine models and pipeline interface
* add pipeline folder structure
* add image matting pipeline
* refine nlp tokenize interface
* add nlp pipeline
1.add preprossor model pipeline for nlp text classification
2. add corresponding test
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8757371
* new nlp pipeline
* format pre-commit code
* update easynlp pipeline
* update model_name for easynlp pipeline; add test for maas_lib/utils/typeassert.py
* update test_typeassert.py
* refactor code
1. rename typeassert to type_assert
2. use lazy import to make easynlp dependency optional
3. refine image matting UT
* fix linter test failed
* update requirements.txt
* fix UT failed
* fix citest script to update requirements
3 years ago
0a756f6a0d
[to #41402703 ] add basic modules
* add constant
* add logger module
* add registry and builder module
* add fileio module
* add requirements and setup.cfg
* add config module and tests
* add citest script
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8718998
3 years ago
mulin.lyh
feiwu.yfw
wenmeng.zwm
zhangzhicheng.zzc
yuze.zyz
yingda.chen
dangwei.ldw
hemu.zp
bin.xue